Hey there! In regards to your inquiry of James' life in Norfolk, I'd like to follow up with a question or two. Before I get to that, I've looked into James' information and located a few documents which I'm sure are already present in your research. An 1841 England Census containing James' family, an immigration record for James to New South Wales, and a marriage record showing that James married Margaret Enright in New South Wales. After locating those records, I tried to find an 1851 England Census Record with James' information in it. I suppose that record would have shed more light on his life in Norfolk before he immigrated. However, I couldn't find it.
In attempting to find more information on James by exploring other avenues, I looked at his mother. So now comes the question! I happened upon some information that Susannah Frances Wardle's actual birth name was Canfor. The name Wardle came from her first husband, John, in which they had one child, John Gilbert Wardell. Her parents were William Canfor and Anne Canfor (born Steel). So then the question is, was this information already contained in your research? If so, GREAT! If not, then this information may be useful in forming a more complete understanding of the family.
I did find a 1851 England Census with Susannah and her children, but James was not listed in it. Considering that James' Father was a sailor, it may be probable that James took up that trade and was out at sea which may explain why I cannot find an 1851 Census for him. Thus, with all the information that I have at the moment, I'd like to conclude that he may have been out on the sea during those few years before he immigrated to Australia.
I hope these findings proved helpful to your research.
Feel free to ask any follow-up questions.
