Fraser, you are making all sorts of assumptions that you cannot justify from the facts you have.
Something that I have noticed is that depending on who writes down the info depends on how it is spelt, James fraser Rigbys marriage cert. gives his name as Regby!
Spelling wasn't fixed or 'correct' until about the early 20th century. Until then, any old method of spelling could be used and it's not unusual to find the name of one person spelled in different ways at different times on different documents. Sometimes, also, it is quite hard to decipher what the person intended to write.
James Fraser Rigby had a child Maryjane Low(1861-1932) and a paternity suit against him by Christian Low(1839) was accepted by the courts but it seems James had nothing to do with her because Maryjane was listed as staying with her Grandparents George(1804) and Ealenor(1807) Low in the 1861 and 1871 census.
If the court found that James was her father, then you can be pretty sure that he was. It was absolutely the norm for an illegitimate child to live with and be brought up by grandparents, most often the maternal grandparents. The fact that Maryjane was not living with her father tells you absolutely nothing about her parentage. Maybe James' wife refused to take on his illegitimate child, or maybe Maryjane's mother didn't want her to be handed over to a stepmother. You can never know the reason.
I still think James left Fraserburgh with John Fraser to seek work in Edin. or he was avoiding the paternity suit whatever the reason he makes his way back to Aberdeen and settles down with Mary.
Almost certainly for purposes of work.
In 1861 he was in Lerwick with a cousin, using his real name. In 1862 the court noted that he was in Edinburgh. Later that year he married in Aberdeen. If he was trying to do a runner, he made a pretty lame job of it!
An interesting point to note is the Rigbys have used Fraser as either a first name or a middle name in every generation in my tree right down to my son so James Rigby must of thought highly of his Grandparents James and Jean Fraser to continue using the Fraser name.
Giving a child a family surname as a middle name was pretty standard practice from the middle of the 19th century onwards. It's also fairly common for a child to acquire in later life a middle name that isn't on the baptism record or birth certificate.
I cant get a hold of James Fraser Rigbys birth cert. but it is in the census as Ireland N.K. which I'm led to believe is not known, wheather this is not known if he was born in Ireland or not known where about in Ireland who knows.
In 1841 his parents told the enumerator that James was born in Ireland.
Another mystery is why his father and mother (Thomas and Anne) went over to Ireland (during the potato famine) for a few years had James then came back to Fraserburgh???
the 1841 census lists them at Philorth House, Fraserburgh: Thomas Rigby, 20, male servant, born England; Ann Rigby, 20, born Aberdeenshire; James Rigby, 2, born Ireland. (there is also a Jean Rigby, aged 1, born Aberdeenshire, with James and Jean Fraser in High Street, Fraserburgh in 1841). So it looks as if they moved from Ireland to Fraserburgh between James' birth in 1838/9 and Jean's birth in 1839/40.
The 1851 lists James Rigby, grandson, aged 12, in Fraserburgh with James Fraser, 73, retired flesher, born Rathen and his wife Jean, 50, born Fraserburgh. (I would want to check Jean's age by checking the original of that 1851 census, as it it so much at odds with her being 60 in 1841. Or is she a different Jean altogether, a second wife of James Fraser?)
So I don't think there can be any doubt that James was born in Ireland in 1838/9.
The Irish potato famine started in 1845, so James was born well before the famine. Thomas might have gone to Ireland to work, or, just possibly, he might have been in the Army, which might explain why he moved around so much before finally settling in Moray.
The 1851 lists them at Darglans, Knockando, which is in Moray, about 10 miles south of Elgin: Thomas, 33, gamekeeper, born England; Wife Ann, 34, born Fraserburgh; children Ann, 9; Mary, 7; William, 6; Thomas, 4; and Lucy, 2, all born in Fraserburgh. So you know that they moved from Fraserburgh to Knockando between 1848/9 and 30 March 1851, and that if they did go to Ireland at any time during the famine, it must have been for a very short time.