There is a death listed on 09 February 1863. John Bryant, age either 33 or 35, hard to read. That's the parish record says because I'm afraid it was in the workhouse, and the parish to be charged is St George, Southwark.
Saint George's Workhouse, Mint Street: Register of deaths; Call Number: SOBG/035/2.
Slightly more cheerful is a John Bryant's burial in 1885. He was age 58 so that fits but no further details apart from his address, 11 Lower Orchard Street, Clapham. He was buried in Norwood Cemetery Lambeth on 2 May 1885, so if you haven't ruled him out would it be worth looking at him on the 1881 census to see if he could fit your John?
London Metropolitan Archives, Norwood Cemetery, Norwood Road, Lambeth, Transcript of Burials, 1885-1886, DW/T Item, 0943; Call Number: DW/T/0943.