On the 1861, we have:
Fraserburgh 196 ED 4
Alexander Watt, 46, Fisherman, b. Fraserburgh
Magdalene, 31
Margaret R, 11
Charles, 9,
Mary, 7
Alexander, 2
Magdalene, 2
All born Fraserburgh
Margaret would have been b. circa 1850 but is not on the 1851 census with the family you have.