His father would have been born prior to 1800 & the 1841/51 for Durham only has one possibility. Born 1794 Leyburn Yorkshire but surname is Rowland - not Rowlands. Occupation was farm bailiff in 1851. Wife Elizabeth.
In 1871 Joseph's wife is Mary & there are Scott stepchildren in household