It might be worth you asking for help from the Germany bit of the board - to me, Clifford doesn't sound hugely German, so they may have anglicised it when they emigrated.
There is another German Clifford family that I noticed on the census, might be worth keeping a note of them (if you haven't already) because it might match up somewhere. This Clifford family consists of Elizabeth Clifford and her daughter Jo(h)anna who married James Peryer in 1846 (All Saints, Edmonton). Joanna gives her father's name as Jacob Clifford, a shoemaker.......
(1851 HO107; Piece: 1578; Folio: 260; Page:7 name is Peyer
1861 RG9; Piece: 372; Folio: 55; Page: 2 name is Parbery
1871 RG10; Piece: 710; Folio: 45; Page: 6 name is Perrier, but gives POB as Niederwiesel, Germany)