A further thought on your George Coe being born in Sussex, but the family then returning almost immediately to Suffolk.
As it was uncommon for Ag Labs and their wives to travel considerable distances in those days without at least making a stab at setting up a more permanent home, I considered why he may have come to Sussex in particular.
One possibility is that his father was in the Suffolk militia. The Suffolks were certainly based around Brighton during the spring and Summers of the Napoleonic period (and perhaps in the post Napoleonic period too) and it was not uncommon for militia men to bring their wives and even children with them.
So you might try checking out whether his father Samuel was in the Suffolk militia.
Roy G