If James was truthful with the 1911 birth location information of Llandysul, and the two met, then I would assume that it would have been between 1891 (when JGH is in Swansea) and 1895 (when James marries Sarah).
Something significant must have happened with the Swansea family in the 1890s (I assume the mother dying) because 2 of JGH siblings (Ernest & Florence) are living with grandfather John Parker in 1901 and the other brother Brinley (shown as Brindley in 1901) is living with an Elizabeth Cornelius and shown as a nephew. Elizabeth Cornelius nee Parker married William Cornelius in 1881 Swansea.
There is one Mary Harris dying Swansea 1891 age 43 and another 1897 age 50 - both of whom could be the lady. If the mother was the 1891 death and the family split up at that time then JGH could have just moved on.