Dealing with the Alexanders first-
Mary Alexander had a brother Robert who might have been witness at marriage in 1862 but don't have a John Alexander anywhere in family.
Joseph Gilmore did have an uncle, Torrens Alexander, who went to N.Z. in 1875 and had family. When Torrens died 1935 in Taradale his children were in Puketapu, Taradale, Christchurch, Rotorua & Dunedin. On the Alexander side there was also a cousin, John A. Peden, who went to Australia.
Now for the Gilmores-
James Gilmore married Nancy Anderson & had a son, Joseph (likely other children as well).
Joseph Gilmore married Jane Moon. Children-
1. Arthur Gilmore
2. Joseph Gilmore, Philadelphia
3. Jane A. Gilmore, Philadelphia
4. Ellen H. Gilmore, Philadelphia
Arthur Gilmore married Mary Alexander. Children-
1. Emily Gilmore, Philadelphia
2. Margaret Torrens Gilmore, Philadelphia
3. Joseph Gilmore, N.Z.
4. Edward Arthur Gilmore, Glasgow
5. James Gilmore, Philadelphia
6. Jane Moon Gilmore, Philadelphia
7. Alice Mary Gilmore, Philadelphia
If your Joseph did go to N.Z. with a brother then it's one we've not heard of yet. Perhaps the photo was labelled years after it was taken? We do have lots of old photos in family albums (mostly not labelled) including some very old ones taken in Philadelphia but not even sure which side of the Gilmore family they were from. I can't recall any unknown ones from N.Z. but would be interesting to see if any resemblance to others we have.
I do know that your Joseph, and siblings, went to Aghadowey School (on site of present Aghadowey Presbyterian Church hall) then later Gorran School (don the road from present Gorran School) but from that time until he appeared in N.Z. I've not really tried to search for him.