Ok, if we forget the birth date for now, have you found that 'Baby' from his birth through all the census' with the named parents or does he disappear?
I would check this out, his mother/parents may have died & possibly James brought up by your relatives?
That's how I would go about it anyway & if parents haven't died, he's not with them in census' then look for a death, if no death then may be your James?
Annie