Hi Maggie,
If James is illegitimate, I'm wondering if his middle name Leder is a clue, it has the look of a surname, could it be the surname of his father, was his father perhaps a James Leder! It's not unknown for the mothers of illegitimate children to include a clue to the child's father in the baptismal names, might be worth looking for Leder as a surname in the 1841 & 1851 census for Benhall and Halesworth, perhaps even some nearby places.
Regards