So are you saying this 1851 census entry is incorrect?
'In 1851 Thomas was with Sarah and the two children.'
1851 census HO107 1497 folio 63
45 Bayham Street St Pancras  Middlesex   
Thomas Sayer 24  Brighton Head  Married Journeyman Bricklayer
Sarah Sayer 18  Pancras, Middlesex, Wife  Married
Sarah Sayer 10mths  Pancras, Middlesex,  Daughter   
The Kungfu magazine says there was a three year gap between Thomas' daughter Sarah and his son Thomas. The 1861 census does not show either child with Thomas or with Sarah. Sarah's age and place of birth on the 1861 and 1871 censuses is compatible with the details of the 1851 census above.
1861 census RG 116 folio 146
10 Belle Vue Cottage St Pancras  Middlesex   
Thomas Sayers 35  Brighton, Sussex, Head  Married PUGILIST Champion of England!!!
Charlotte Sayers 24 Uckfield, Sussex,  Wife Married Housekeeper 
Peter Sayers 12 Sussex, Son    
There is a Thomas and Sarah Sayers aged 10 and 11, birthplace unknown, boarding in the equivalent of a 'Dame school' in Finchley on the 1861 census. 
1861 census RG115 folio 45
1 York Street St Pancras  Middlesex   
James Aldrige 40 Pancras, Middlesex,  Head  Unmarried Cabdriver
Sarah Sayers 28 Pancras, Middlesex, Visitor Married Mangling
James A Sayers 1 Pancras, Middlesex, Son  
 
1871 census RG10 249 folio 31
34 Peckwater Street St Pancras  London   
James Aldridge 50  London, Middlesex,  Head  Married Cabman
Sarah Aldridge 36  London, Middlesex, Wife  Married   
James A Sayers 11 London, Middlesex, Son    
Alfred A Sayers 9 London, Middlesex, Son    
Charles A Sayers 7  London, Middlesex,  Son  
Robert A Sayers 5  London, Middlesex, Son  
Maria Bond abt 1804  London, Middlesex, Sister Widow  
1871 census RG10 222 folio 39
5 Melton Street St Pancras  London   
Charlotte Sayers 42  Bristol, Sussex, Head Married Tobacconist
William Francis Olrod 45  Sanchall, Surrey, Lodger Piano forte maker
1881 census RG11 199 folio 28
5 Melton Street St Pancras  London 
Charlotte Sayers 53 Busted, Sussex,  Head  Widow Tobacconist
William Olrod 54 London, London, Middlesex, Boarder  Widower Piano forte maker 
Emily H. Sayers 9 London, Middlesex, Granddaughter  
plus 1 boarder
Regards
Valda