Hi
Am picking up a marriage of a Joseph Fisher to an Ann Mumford on the 24 Oct 1808 at Saint Leonards, Shoreditch, London, England (IGI)- but of course, cannot tell if this is the correct one.
There is also a Joseph Fisher married to Ann Culverwell at Southwark, St Olave in 1813 (Pallots marriage index)- these appear to be the most likely at this stage.
Is this your Charles?
CHARLES BROWN FISHER
Birth: 14 MAR 1817
Christening: 04 MAY 1817 St Botolph Without Aldgate, London, London, England
None of the others are showing on the IGI.
On the 1861 census there is a John Fisher born 1819 in Islington a brickmaker and wife and family,
There is an interesting one in Cadogan Squm Chelsea, St Lukes.
who was born 1818 - property houses is listed as occupation but what caught my eye was the Sarah Culverhouse neice living with them - as in connection with the Ann Culverhill who married a Joseph Fisher - ? very similar names - a co-incidence?
Unfortunately all I am giving you is more questions to be followed up.
Am not picking up Joseph & Ann on the 1851 but I only have a partial index - someone else might be able to help.
Was Charles working?

in Essex when assume got himself into trouble - or you have no idea?
Cougar