This family are at 5 Bargeman's Row Swansea in 1871 with lots more children
William Williams Head Male 39 1832 Glamorgan, Wales
Mary Williams Wife Female 37 1834 Glamorgan, Wales
William J Williams Son Male 17 1854 Glamorgan, Wales
Mary A Williams Daughter Female 16 1855 Glamorgan, Wales
Ellen Williams Daughter Female 14 1857 Glamorgan, Wales
Evan Williams Son Male 12 1859 Glamorgan, Wales
Jane Williams Daughter Female 10 1861 Glamorgan, Wales
Elizabeth Williams Daughter Female 7 1864 Glamorgan, Wales
Thomas Williams Son Male 5 1866 Glamorgan, Wales
John Williams Son Male 2 1869 Glamorgan, Wales
Lucy Williams Daughter Female 1 1870 Glamorgan, Wales
The obvious here is Thomas, aged 5/6 years younger than we expect him to be from newspaper article?
My 1881 pick would be the family at 25, Graig Terrace, Swansea (Town), Swansea
so, I'm not so certain now, maybe this is your man
