Thank you for info above - I had found that 1851 census info and I think it might be the right family for my Rafferty line. But they are all listed as born in Ireland and not sure where in Ireland so can't really do any further searching back without an Irish county/place to look into.
I wondered if there's somehow a connection with that Martin/Emily Rafferty family but still trying to piece everything together!