Just been looking through at we have for James in respect of the 1871 - 1901 censuses. He is pretty consistent overall with his age (except for 1871).
1871 - b. 1832 in Hogganfield
1881 - b. 1841 in Barony Parish
1891 - b. 1842 in Glasgow
1901 - b. 1842 in 'Hoffonfield', Lanarkshire
I saw this entry in 1841 for example:
Widow Graham 45 b. England
James Graham 10 apprentice blacksmith
Address: 29 Brown St, Glasgow Brownfield
Hard to say whether this could be James at this stage isn't it.
Would be good to find some trace of him before his marriage on the the 1841-61 censuses

Confused on a detail on the reporting of James' death. Was it William his son who reported the death or an Uncle James Graham? You have mentioned both. Or do you mean a possible Uncle James Graham could have brought up young James and could explain the confusion on names?
Monica