Not having much luck with this family. I don't think you'll find them in 1891 as 10 year old Tom's unclear birthplace on the 1901 census definitely has the word America followed by BS British Citizen. There was a census in the US in 1890, but most of it was destroyed in a fire.
Think this might be grandmother Mary in 1881
Croes Ciliog (few households away from Hector Inn) Garnvach, Nantyglo RG11/5420 folio 26 pg 46
Mary Williams head widow 80 b. Carms, Llandovery
haven't spotted any of the others as yet in 1881. From the entry in 1861, you can see the inconsistencies in age and birthplace - that doesn't help when the surname is Williams

11 Little Croes Ciliog (next to Hector Inn) RG9/3997 folio 76 pg 58
Mary Williams head widow 35 b. Carms, Llandovery
John son 27 coal miner b. Govilon
Elizabeth dau 23 a. Aberystruth (deaf and dumb from birth)
William son 16 b. Hereford, Clifford
Lewis son 14 coal miner b. Aberystruth
Now, Mary can't possibly be 35 but that's what it says on the census. The parish of Clifford is only a few miles from Hay.
Still not finding them in any other year!