On a line I'm researching at present there's a chap who was a warp dresser in a Lancashire Mill, but at about the same time his son, marrying a long long way away in London, describes him as a "gentleman" From wife-to-be's name and family, yes, he was marrying well above his station! Wonder what happened if the in-laws ever met?
A "Servant" could be very humble status, or at a bit higher level or rather prone to ape the gentry he / she served, in some cases - one woman I knew who had been lowly in service became in later years all fish-knives-and-napkins, and more obsessed with "correct" ways than Hyacinth Bucket! (Not implying your chap was at all like that, but if via marriage he'd come to a level where he didn't need to work, then he could describe himself rightly as being a gentleman [of leisure], then, couldn't he?).