In 1861 Charles is transcribed with surname Leslie on ancestry. Parents are Lester.
I wonder why his surname was written out as his mother just has ditto.
Following on from your comment I did a quick search last night for a Charles Leslie born c1830
this one came up on 1881 census in Warwickshire his occupation Accountant, wrong pob though
I haven't had a chance yet to follow this family up to see where it leads - if anywhere
https://www.familysearch.org/ark:/61903/1:1:Q27N-V2G8?lang=en (no similar records were found)
Charles Leslie age 51 Spouse Sarah E.
They have six children, one of those is named Edward Clerk (out of employment) born c 1863 Birmingham. First child named John, Clerk (Accountant) born c 1861 Broughton In Furness, Lancashire
If I get chance will try and follow up later 
Here's what I've found for the above
Charles LESLIE on 1881 census - looks as if his surname should be
LEESE
1901 census surname transcribed as
HEESECharles 71, occ
manufacturers Clerk , Sarah M 71 & Florence E 35
1871 census surname
LEESCharles 40
Accountant Sarah M 40 Birmingham Wife
John 11 Broughton Son
Edward 8 Erdington Son
Millicent 6 Daughter
Florence 5 Daughter
Jessie 3 Daughter
Ellen 15 Niece
LEESE,
EDWARD ERNEST
mmn JEROMEGRO Reference: 1862 S Quarter in ASTON Volume 06D Page 295
LEESE, MILLICENT ADA mmn JEROME
GRO Reference: 1864 J Quarter in ASTON Volume 06D Page 348
LEESE, FLORENCE ELLEN mmn JEROME
GRO Reference: 1866 M Quarter in LANCASTER Volume 08E Page 623
Marriage Sept 1851 Kings Norton same page
Charles
LEESE / Sarah Maria
JEROMEMarriage image on Anc*y no father's names given,
Married 28 Aug 1851
Charles occ
coach maker,
one witness
Rebecca LEESEAs your Grace Emily & Charles LESTER were on the 1871 census in Durham I think you can cross this Charles off the list