There is also a Thomas Selby in Missisquoi County in 1851 (also with a Hannah aged 70 years) Could this be a brother of Joseph and Hannah (possible mother) is listed twice - once with Joseph and once with Thomas

Thomas Selby aged 42 years born England, Abigail G H Selby aged 40 years born England , Hannah Selby aged 70 years born England, Hannah aged 13 years, Jonathan aged 11 years, Mary aged 8 years, Joseph aged 3 years and Thomas Selby aged 9 years
Thomas Selby (farmer) still listed in the Eastern Townships Directory and Gazetteer, 1875-1876 for East Dunham (not to be confused with the Thomas Selby born 1811 Yorkshire who passed away 1889 - Verulam, Victoria, Ontario, Canada)
Abigail is widowed in 1881 Missisquoi Dunham Canada.
Joseph Selby 31
Abigal Selby 69 Born 1812 England. Widowed.
Hannah Selby 41
Sandra