I wonder if this Joseph is the one living with Mary and 2 other children in 1841:
Brigham Row, Crosthwaite, Cockermouth, Cumberland:
Mary Harrison aged 25 (born c.1816)
Thomas Harrison aged 12 (born c.1829)
Henry Harrison aged 9 (born c.1832)
Joseph Harrison aged under 1 (born c.1841)
In 1851, they are still living in Brigham Row, but this time with Margaret Harrison, unmarried, born c.1791, who is the head of household. Thomas and Henry could be the youngest children of Margaret, but Joseph seems too young to have been - Margaret would have been 50 years old when he was born - so could he have been unmarried Mary's illegitimate son? They are all described as born in Keswick, Cumberland, but logically Joseph was probably born in Cockermouth, where he was living as an infant.
Queenie

EDIT: David beat me to it!