have you seen the baptisms of Justine & William? The address given for Joseph & Kate on Justine's baptism is given as Kilburn Fields (she was baptised the same day as a daughter of Levi Boswell). On William's baptism in 1876, the address is given as 4 Latimer Road, near Wormwood Scrubs. So, that's a couple of records from before 1891.
Joseph may not appear with James & Phoebe on another census, but Phoebe is on the 1851 census, father must have given wrong age on the 1861 - her baptism was 3/3/1850.
1850 Crockets Court, Hampstead HO107/1492 folio 241 pg 30
James Redworth head mar 32 labourer
Elizabeth wife laundress b. Willesden
James son 11
Fanny dau 9
Mary dau 6
William son 4
Phoebe dau 1
all born Hampstead except Elizabeth
He also appears on the 1871 census with James who has another wife
9 Circus Road, St Pancras RG10/242 folio 19 pg 29
Ja
s. Redworth head mar 53 labourer b. Hampstead
Car wife 46 laundress b. Sussex Rye
Jos. son 15 labourer b. Hampstead
Laura dau 14 b. Hampstead
Car dau 10 b. Hampstead
Ellen dau 2 b. Kentish Town
Car Redworth snr shows up as Caroline in 1881
