Cant see them anywhere together as a family before 1901.
A possibility.
1891 census, Laundry, Bettws Bleddrws, Cardigan, Wales. RG12/4553 p6.
Frederick Smith, head, 54, Coachman Dom Servant. b. Durham
Emma Smith, wife, 48, b. Belfast,
Robert Smith, son, 13, b. Canterbury, Kent,
Joseph Smith, son, 11, b. Shorncliffe, Kent.
Emma and Joseph are both right age, both born right place and have a Durham connection. Its possible that Frederick died and Emma moved north and met James, or was already there.
Theres also a birth of a Joseph Henry Smith on FreeBMD in Elham, Kent (includes Shorncliffe area), Sep qtr 1879, vol 2a p 963. The only Joseph Smith for a few years either side, that I checked, so the middle name matches also.
Alan.