Hi Thomas2Sam :-)
I'm also trying to trace a James Dillon, of Co. Armagh, and I'm just starting to wonder if they're connected. I'm wondering if yours could have had a first wife, Judith McCavney/McKeavney, and at least a couple of children, Isabelle b. 1845, Richard possibly b. 1844.
It's just ringing bells, because Richard was living in Greenock, married there in 1865, but was from Armagh - I think he stayed there, but I have lots of matches with Australian Dillons, and his daughter, my great-grandmother Margaret, eventually left New Zealand and went to Sydney. Her mother was a McCosker, who died young, and many of them also went to Australia.
Have you done DNA testing? I've got a large Dillon tree mapped out that connects my Dillon matches, but I'm still looking for my link.