Hello Betsy,
Wonder if James was "out of the country" and missing the census - looking at the census James Coulter of Ballantrae was in Kings Dock. Liverpool (vessels) unmarried in 1861.
There are a few entries on NSW, Australia Passenger Lists which would fit his profile but not definitely distinguish him - James Coulter, aged 23 years from Scotland, AB, travelling London to Sydney, 31 Jan 1866 on the SS Lochiel
James Coulter. Age 26 years. AB from Ayrshire on the Isle Of France - 14 Dec 1869 - Hong Kong to Sydney. NSW
another 29th July 1870 - James Coulter, aged 27 years, AB from Ayrshire - going London to Sydney on the James Aitken
Sandra