The members of the Inn, lawyers living without women, employed a small army of servants who lived on site.
The Inn maintained a charity for poor children and those were regularly laid opn the doorstep. A childrens home was run, with women staff. The children were brought up, in turn, to become Inn servants.
The most senior of the servants were very important and well-off, such as the Senior Butler.
So, no - your ancestors did not own the Inn and were probably servants there. About the time you refer to there was a strong Puritan movement which included many lawyers, some of whom were also clergy. Some of those emigated to New England and Pennsylvania particularly and might have taken their servants with them.