1871 Census shows them living at Williamsfield, Holywood which you have already.
1881 - have not found them
1891 - living at Riddings Cottage, Irongray, Kirkcudbrightshire:
Joseph, Elizabeth, James, Agnes, Mary you already have - William is not there.
In addition:
David Wright Telfer b abt 1878, Dunscore,
Alexander Telfer b abt 1888, Irongray, Kirkcudbrightshire,
Elizabeth Telfer b abt 1889, Sanquhar, Dumfriesshire
1901 - living at Knowehead Cottage, Holywood.
Joseph, Elizabeth, William (Estate Carter), Mary (worker on estate), Alexander & Elizabeth (scholars)
Hope this helps.