If James is on the 1841 but not on the 1851 then i reckon the most likely explanation is that he died between those dates. Unfortunately Statutory records were after 1854 which can make confirming this harder.
Alternatively there may be another explanation such as him out working or at relations at the time of the 1851 census.
Robertson is one of the most common surnames in Scotland and the same can be said of James and Janet as forenames back then.
So the more details you provide the easier it will be for anyone to be of help to you.
If you have the family on the census for 1841, 1851, 1861 then please tell us all the family members names ages exact birthplace etc.
Did the whole family move to New Zealand or did anyone else stay behind?
Do you know the mothers maiden name?
I've found some info on IGI LDS, do any of the following fit your family?..
m. 19 Jul 1812 James Robertson and Janet Stewart
b. 1821 John Robertson
m. 14 Jan 1815 James Robertson and Janet Moncrieff
b. 1817 James Robertson
b. 1819 Ann Robertson
b. 1821 William Robertson
m. 30 Apr 1832 James Robertson and Janet Maclaren
b. 1837 James Robertson
b. 20 Mar 1841 James Robertson
b. 1844 Catharine Robertson
m. 06 Oct 1833 James Robertson and Janet Seaton
b. c1834 Donald Robertson
b. 26 Dec 1835 James Robertson
b. 1838 Mary Robertson, Margt. Robertson
b. 1846 Jane Robertson