There are 3 male Patterson births registered in Downpatrick in 1864 (Vol 11, page 452, Vol 11, page 456 & Vol 16, page 479). The births were registered with no forename, as was common at that era. Any of them might be the James Pentland Patterson whose 1864 death you have found.