yes you got the right man!!!
I've tried looking on the 1851 census for a Mary James born c1829 in Neath. There didnt seem to be one living in Neath. Interestingly there is one living in Merthyr, born Neath, married to a Thomas James, she was age 19, no children. I've had a look on free bmd marriages and in Sept 1/4 1849 Neath theres listed a Thomas James and also a Mary Morgans.
I've put those details in again for 1861 and there doesnt seem to be a Thomas & Mary James with her details. So what happened to that couple? On the 1861 I've got my Mary Jones nee James living with Richard. Had very quick check on deaths and between 1851 and 1853 (when first child of my relatives was born) and there are a few deaths of Thomas James in Merthyr... I'm wondering if either of those Merthyr marriages in 1851 & 1852 between Richard Jones and Mary James show Mary to be a young widow??? That may be a clue to me getting the right one, or maybe I'm clutching at straws! I dont know!!!