Here it is on an 1887 6in OS map
https://maps.nls.uk/geo/explore/#zoom=17.1&lat=51.05763&lon=-3.57813&layers=257&b=osm&o=100The underlying modern map has it as Draydon Cottages but on the current OS map it is labelled as Draydon Farm
In the tithe records of 1839 it is named as Draydon Farm (but marked as Draydon on the tithe map) and the plots add up to 216 acres, so a little smaller than the 1851 census indicates (260 acres). The owner is the Reverend George Ansley and the occupant is John Webber.
I found John Webber at Drayford in the 1841 census (John Webber is a remarkably common name at that time in the county!). He is 65, a farmer and his wife Mary is 70.
Added: There is a death record (Somerset Archives via FindMyPast) for the burial 20th Jan 1848 of
John Webber of Draydon in Dulverton age 72, so I suspect this is when the farm passed to the Follett family.
For anyone following this up, be careful to not confuse this John Webber with another John Webber in the area of a similar age who fought in the Peninsular War, but survived until 1868.
Added again: I just realised that the owner’s name in the tithe records is probably Anstey, not Ansley, connecting back to the original question. So the farm was still owned by the same family, but rented out.