I had hoped to find some family member on the 1841 Scottish census but that's not going to be possible. Unless any of the the family returned to Scotland/England after he died in which case they would only be on the 1861 onwards
I have checked the English and Scottish 1861 and there are no Imray's born overseas. Did he have a son Alexander born about 1829?. There is an Alexander Imrey aged 32 - birthplace unknown - boarder in Suffolk - occupation groom in racing stable
The 1871 has an Arthur J Imray born Sri Lanka 1858 at a boarding school in London who may be a grandson of Robert