Have you confirmed his father's name was John from either of his marriages?
Are you basing the information about his father on the 1841 census? The ages don't actually fit for John from the 1841 census.
On the 1841 census that John Miller junior's age is given as at least 20 (
adult ages, those over 15, are usually rounded down to the nearest 5 on the 1841 census
) therefore birth year of circa 1821. The John Miller you are interested in, shown from the 1861 census onwards, was aged in 1861, 31, birth year circa 1830, from 1871 onwards in Edmonton, 1871 aged 42, birth year circa 1829, 1881 RG11 1390 folio 4 aged 50, birth year circa 1831, 1891 RG12 1082 folio 155 aged 59, widower born Kennington, birth year circa 1832, 1901 RG13 1263 folio 125 aged 69 birth year circa 1832. So very consistent about his age.
I haven't given any information on John's second marriage. He was married to Elizabeth on the 1901 census, so a marriage sometime between 1891 and 1901.
The 1861 census entry I've given for Charles Miller in Lambeth, chimney sweep, does fit the age of the Charles given on the 1841 census entry, aged 12 birth year circa 1829. 1851 census HO107 1575 folio 438 aged 21 birth year circa 1830, 1861 census aged 32, birth year circa 1829, 1871 RG10 675 folio 10 aged 42 birth year circa 1829. All Lambeth and a sweep. By 1881 Elizabeth was a widow RG11 606 folio 11.
Lambeth was south of the River Thames so Surrey not Middlesex.
Regards
Valda