You see even though its hard to trace gypsies back the saving grace is that they dont lose each other like other people as they are such a close nit community and i would have money on it that all the names you have mentioned such as castles , hiltons, lee's, taylor's, penfolds, scott's , coopers , keffords, are directly related to the people that surround me today and my guess is that because it sounds like a couple of these gypsy women married gorjas (none gypsy people) like you family the Lawrance's back then that would of been shameful meaning these women would of been expelled for the community leading to over time losing all history of where they come from .........this is just my theory but I would have money on it and this would explain why Job and Jim held the name Castle as they considered there self's gypsies