Hi,
From the very poor description given by the enumerator of the direction and distance he walked in 1841 & 1851 Census it is very difficult to discover the actual spot.
However Callow Park (still there today) is mentioned as is Sitch Farm
So you are looking in the very general area of Callow Park, Derby Road Wirksworth DE4 4BN
situated on the Wirksworth to Duffield road B5023.
You could also contact John Palmer on the Wirksworth site, always very helpful, the connection is:-
www.wirksworth.org.uk his contact details are in the middle of this page.
If you visit the Derbyshire Record Office, there is a Callow & Kirk Ireton Enclosure Map of 1807 which will probably show the exact site of the Mill.
Reference for this is:-
D258/1/15/2-3 1803-1807
Spendlove