I found the 1851 census (Kirk Smeaton, Pontefract) for:
Charles Neal (c.1811) born Bow, Middlesex
Mary (c.1811) Wood Bridge, Suffolk
Sarah Ann (c.1839) Kirk Smeaton
Henry (Bridsy?) (c.1841) Kirk Smeaton
Thomas (c.1844) Kirk Smeaton
Frederick (c.1846) Kirk Smeaton
Charles (c.1848) Kirk Smeaton (probably your gggrandfather)
Alice Mary (c.1850) Kirk Smeaton
and also in 1851 (Gringley, Notts)
Charles Brewitt (c.1820) Gringley
Mary (c1824) Gringley
Elizabeth Ann (c.1849) Gringley