Another thought:
1851
Land of Oldwhat, New Deer
William Chalmers, 33, farmer of 140 acres arable. b. New Deer
....
.....
Margaret Fraser, serv, unm, 20, House servant, b. Fraserburgh
The other two Margarets showing on the 1851, b. Fraserburgh don't seem to fit.
1861
Aden Arms Inn, Deer
Jane Fraser, wid., 52, innkeeper and farmer, b. Strichen ( this looks like George's widow?)
...
....
Margaret Fraser, serv, unm, 31, waiting maid, b. Fraserburgh
Could Margaret have manipulated her age to bring her more into line with James Imlah's age (both 26 on marriage cert). If so, then the 12 year old Margaret with Christian on the 1841 might be her!
Gadget