Hi,
I wonder if this is your Joseph on the 1861 census:
Baladen Brook, Tottington Higher End
John Taylor, Head, Wid, 63, Ag Lab, Bury
Ann, Daur, U, 23, Cotton Factory Worker, Bury
Nancy, Daur, U, 21, Cotton Factory Worker, Bury
Milicent, Daur, U, 19, Cotton Factory Worker, Bury
Margaret Ashworth, Daur, Wid, 34, Cotton Power Loom Weaver, Bury
JOSEPH Ashworth, Son, 13, Cotton Factory Worker, Bury
John Ashworth, Grand Son, 11, Cotton Factory Worker, Bury
RG9/3059 Folio 29 pages 7 & 8
Mo