I have family on my husbands tree living on the common at Warbleton in the 1841 census.
I had just thought that addresses in the 1841 census were rather vague ie not 56 High Street as you would see today.
I also have people at "the village" in a Dorset 1841 census - also vague

My husbands family appear to be living 3 doors away from your family and they are all Ag Labs.

In other words, I don't think the address signifies that they were romanies.

HL