I now find he married Harriett in Colwall in 1849, but he is missing in 1851. (He had been arrested and was in prison)
Maybe he was away from home or maybe just maybe this is the window through which he was deported to Trinidad.
I think a visit to the records office is due. I still await a reply from a distant relative of Joseph, in California, who is doing some research for me.