This could be him
1871 Census for Newport, Wales
41 Baldwin Street, Newport
James Lee, 44, Labourer born Bristol
Elizabeth, wife, 44 born in Bristol
Henry, son, 15 Engine Cleaner born Newport
1881 Newport
44 Capel Crescent, Newport
Elizabeth, now widowed. a Housekeeper born Bristol
William, son, 28, Engine Driver, born Newport
Ann Harman, married daughter
Henry was found to be married and is a Loco Driver (Wales 1881)
Alf