James Elson is my GGG grandfather, he moved to London and changed the spelling of his name to Hilson. His son by Ann Young Brown, Robert Henry Hilson, was a tobacconist, and his sons (one of which, Francis, was my great grandfather) moved to Brighton in Sussex, becoming publicans. All quite traceable on the censuses!
I'm trying now to track sideways and back in the Tiverton roots: all I have to go on is James' parents, John Elson and Mary Hodge marriage on 5 Feb 1780 at St Peters Church, Tiverton