There is an 1861 entry for a John Graham, aged 67 and a grocer with his own shop. Showing by himself. Marital status not included (pity!). He shows as born in Stapleton. This entry shows in Walton, Brampton. You could check to see if he shows earlier and then eliminate him as you have your John with wife Jane and family in 1841 and 1851.
Monica