Ok I don't know if by chance I struck a breakthrough on Thomas Duffy - just a guess but I don't know (head's in a spin) whether it all connects.
I found a death record on 29th July 1836 (burial) for a Thomas Duffy in St.Patrick, Lancashire. Not sure if it's related, as any census entries post this (1841) Theresa Duffy doesn't have any Husband listed.
So, I thought, about 40 years back for a birth record, perhaps in a similar area, (and had to include Ireland, of course)
And stumbled on a record (book entry) of a baptism on 15th May 1797 at St.Mary's, Lancashire. With Edward & Elizabeth Duffy as parents. There's another listing in Dunshaughlin in Meath, but i'm not sure whether, either, or any is correct at this stage.
James Hoy - he is listed on the christening record as the father. (I did shoebox it on my Ancestry account.)