I don't know of a Milfordton in Somerset, but it may be Milverton. You can find transcriptions of its registers here
http://www.pbenyon1.plus.com/H_m_w/Mil/Baps/1781-1790.html
There's a Mary Smith baptised in 1789, to Robert and Betty.
But that would make her nearly 50 when William was born, so probably not the right Mary.
In the 1861 census she's at the same address as in 1851, (with Maria Carrington who is now widowed). Her birthplace is given as Bath, age 65! With such a range of birthplaces and ages in the census and such a common name it's going to be difficult to take her any further. This could be her death:
Bath Death indexes for the years: 1870
Surname Forename(s) Age Sub-District Registers At Reference
WADMAN Mary Ann 80 Lyncombe/Widcombe - First Series Bath LW1/21/174
putting her almost old enough to be the one from Mlilverton. This is probably William's:
Bath Death indexes for the years: 1844
Surname Forename(s) Age Sub-District Registers At Reference
WADMAN William 52 Lyncombe/Widcombe - First Series Bath LW1/5/102
David