You need to get a look at a copy of the Register of Electors and/or a copy of the Valuation Roll.
The first will tell you the names of all the people registered to vote as residents at that address, and the second will tell you who owned the property and who the tenant and, if different, the occupier was/were.
There are copies of the RofE and VR in the National Library of Scotland, the National Archives Records of Scotland and probably Aberdeen Archives. There are also likely to be copies in some local libraries in Banffshire, and in the Moray Local Heritage Centre in Elgin.
