Speculation maybe, but useful all the same.
Especially the part about the Harris who was a steward, as this is the very occupation James' second son George favoured!
James was a Hostler, Coachman and latly a Dairyman, but generally called himself a servant in censuses.
My ancestor George was a Butler by the age of 23 (Bath), a ships steward (Southampton), restaurant waiter (for a few months in London), dairyman & assurance salesman (briefly back in Somerset), valet (London)!
I would certainly be interested in exploring those Harris' further, if you have a link please. I find the Ewyas Lacy site rather difficult to navigate.