I've had another look at the accompanying notes and it says:
"Persons, including Children (of whatever age) actually in the Parish, Township or Place at the time
of taking the Account, exclusive of Men actually serving in H. M.’s Regular Forces, the Militia;
Seamen in H. M.’s Service, or the registered Vessels. (this is expressed in the transcription as, for
example, 1m, 1f, which of course means, one male, one female)."
So it is possible that he was at sea on a boat on census night (equally, he may have died by then - this is what's unfortunate about early censuses - if it was 1851 it would say 'married' or 'widow').
I've got an Essex Ancestors account - I've also got the CD of Brightlingsea parish transcriptions that ESfFH have done. Have you had a look at those? Actually - I've also got ESfFH's CD of transcribed memorial inscriptions. I'll have a look and see if the Harveys are on there.
Have you had a look on
http://www.freereg.org.uk/ ? I have transcribed several parishes in the Tendring district, so it's possible the Harveys appear there (a quick search reveals a lot of Harveys in St Osyth and surrounding parishes). There are also Harvey families further along the Colne in Wivenhoe, Fingringhoe and East Donyland, so James could conceivably have come from there.
You might also want to look at ESfFH's transcription CDs of Colchester. It isn't cheap buying the full coverage (1752-1851) but if you have a lot of family from Colchester, it's definitely worth it!