Any clues as to possible sibling ? Any Frasers amongst the 1806 marriage witnesses perhaps? 
What about the other Frasers in Portsea? There was a David Fraser born around 1785 who had a son Roderick. David Fraser was a shoemaker, and Roderick was a bootmaker in the 1841 census. Could he be a brother, or a cousin?
Roderick had a son David  born 1818. 
Also there was a surgeon named William Fraser in Reading in 1851 who was born around 1795 in Nevis.