Hi Marilyn,
Thank you for the information.
I found James on Moray Libindx, ref NM033462. This is where I found the information that he had died in 1924 in Vermont, U.S.A. The same record also states that James is the father of my 2 X Great Grandfather Alexander Paterson or Smith by Isabella Smith. I got in touch with Moray Libindx to ask where this information had come from and they tell me they didn't keep the details. Kind of strange really when information of all sources really should be kept in family history research.
I then got in touch with Vermont State Archives who were unable to find a death record for James (unsurprising if he died in Boston, Mass). They did however send me a whole load of records, including his wife Isabel's death record, Their son James Robertson Pat(t)erson's birth and marriage plus some of his children's birth records.
I have had a look for the Elspet you mention and believe she may be the daughter of James Paterson and Magdalene (Medley) Bremner and was born on 31st August 1819 n Rathven, Banff.
Amongst other children James & Magdalene also had a son James born 31st July 1815. I believe this James may be the father of my James and would also fit in with James being brought up by his Aunt Eppie (Elspeth). Elspet was married to Alexander Simpson and I have found the census records for them, also Elspet's death record.
Interestingly in the 1861 census Elspet (who was widowed) and family were living at Longfolds, three houses away from Isabella Smith and her parents in Wester Darbreich, Rathven, Banff. On checking a map it shows these two properties are very near neighbours. Alexander was born four years later in 1865.
In the 1851 census it shows James living with Elspet and her family at Longfolds and his birthplace is given as Rathven, Banff.
I have a wee book entitled Banffshire The People and the Lands, Part 6 The Parish of Rathven 1750 - 1850 which contains kirk Session Records. One of the records I have found in this book is dated 1844. There are two entries, the first one says that Helen McCraw is with child by James Paterson, the other says that James Paterson is the Father of a child by Helen McCraw. It doesn't actually say that they were "censured for antenuptial fornication" - as it does with most of my other ancestors mentioned.

I think if the rest of the family classed James as a black sheep this was a case a pot, kettle and black.
I would be very interested in learning more about James' half-brother George that you mentioned in your first post.
Lynn