Yes I think you're right, sometimes we see what we want to see

I got confused with my 1851 fellow Ambly who was also a plasterer (and not who I was looking for either!)
The reference for him in the 1841 census is HO 107/734/3 p14. Several of them are transcribed incorrectly, although I'm not surprised as the writing's terrible!
I think I may have also found him in 1851 (but still a bit of checking to do). There is a John Mott living in James Street, Westminster with a wife and son. He is described as a 'servant out of place', which is interesting as he wrote to the 2nd Marquess over a period of several months after he lost his job as he couldn't find a new one without a reference. If this is the correct one then he must have really struggled to find something new.

The reference for 1851 is HO 107/1840 p[512 I think]. This one was born in Mitford, Sussex, so it's something to go on anyway.
Many of the servants 'lived in' in the 1840s, but not all. Those that didn't lived nearby and it was usually the upper servants (eg the chef) who didn't. Although as he had a family he may have had his own lodgings as well.
Vicki