Hi Lammaris. As you say, the censuses only take you back so far, and you then have to look at parish records (baptisms, marriages, burials), which are in the Gloucestershire Archives (and maybe on FindMyPast). As you probably have the Lammary's residents in 1841, work backwards from then through the church records for them. Have you tried to find their gravestones in the churchyard?
The Ley Mary Farm doesn't appear on the Sherborne Estate maps from 1825, so it wasn't owned by Lord Sherborne then. You could also look through the online catalogue for the Glos. Archives for records that mention the farm.
Might I suggest that you give a copy of your findings to the local history society?