Hi Calcorn
That 1851 census entry that Portybelle mentions looks pretty likely to be your James with parents. Unfortunately, there doesn't seem to be an entry showing for his birth/christening on the Old Parish Registers (nor the Roman Catholic registers), in this period before the start of official registration from 1855.
I think, like you say, it is likely his parents may have died whilst he was young. There is an entry for example in the 1861 for 10 year old James Barrie who shows as a boarder at the McAllister household in Old Kilpatrick Dumbartonshire. There are other young boarders showing there too, also from Glasgow.
With possible early deaths for his parents, James may well have got confused about his mother's name in later years, not surprisingly. However, I would probably look to the Poor Relief documentation and the confirmation of his mother's name as Elizabeth Brownlie. The Poor Relief Board would have carried out numerous checks before the granting of poor relief to James in his later years.
Is there anything in the names he gave to his own children that might give you more clues about his own family?
Monica
