Hello Fay,
I can't help you specifically with your family, but I can give you a general view of SE London around that time. Going by the bits of the 1871 census I have glanced through, there was certainly a substantial Irish community in Woolwich.
As far as the Romany connection goes, I found these Lees in the 1871 census in Eltham, not a million miles from Woolwich:
Caravan on Eltham Common
Belcher Lee, head, 37, hawker, born Kent Hollingborne
Jane Lee, wife, 37, hawker, born Suffolk Coling(?)
Mary Lee, dau, 12, hawker, born Kent Maidstone
Regards,
Phil