So now I am totally confused.
I don't think it's me that is confused. There are a number of trees on ancestry which have Thomas Adsley Brown as the son of Thomas and Emma. This seems to be due to a spectacular failure to do proper research.
Well there was an Emma Adsley b 1853, Elham
But did she marry Mr Swain?
Emma was baptized at Elham, 20 Feb 1853. Father Stephen was a labourer.
Emma Hasley married Stephen Edward Swain, 28 April 1873 at Paddlesworth. Her residence was Elham, father Stephen, a labourer.
She said she was of full age, but I don't think there's much doubt that it's the same Emma.