Not sure if this is worth considering but here goes.
Going back to at least the 1200s some Scots nobles also held lands in England from King Edward of England, so there is a possibility of an interchange of personnel. I would imagine that when Scotland kicked the English out, Scots nobles with lands in England would be faced with losing either their Scottish lands or the English lands, any that chose to lose their Scottish land in favour of keeping the English ones would no doubt have taken people with them back to England.