Hi Kirsten
Have found a possible entry for your Samuel on the 1861 census (the only one I could find born in Dumfries):
Residence: 31 English St, Dumfries.
Head: John HENRY
Born: abt 1810, Ireland
Occ: Freestone Broker
Spouse: Alice
Born: abt 1826, Ireland
Son: Samuel
Born: abt 1851, Dumfries, Scholar
Daughter: Alice
Born: abt 1854, Dumfries, Scholar
Hope this ties in something else you have.
Annie