Can't spot him easily in the censuses - do you have the references (Year/Piece/Folio/Page)?
Could this be him?
1891
Yew Tree Farm, Heap, Bury, Lancashire
Thomas Scott Head M 36 Farmer Scotland
Elizabeth do Wife M 33 Scotland
William do Son 9 Scholar Scotland
Drummond do Son 8 Scholar Scotland
Thomas do Son 3 Lancashire Heywood
Agnes do Daur 1 Lancashire Heywood
Jessie do Daur 2m Lancashire Heywood
Margaret Queen Serv S 18 General Servant Scotland
RG12/3132/37A/36
GF