Yes, you are right in the 1950s/60s it was considered polite to call someone 'coloured' and not polite to say 'black', and to say 'Negro' was if you like a technical term for someone of African ancestry. And today 'black' is the most accepted word and the others are not used. Simply things change over time.
None of which helps with the meaning of the word in the early 1800s. And that Wikipedia article also says nothing about the use of word in England at that time. So until someone come along with a reference, we still can't be sure what was meant by the term, though I am sure it could easily have meant someone of darker skin than the usual Englishman.
And as for your own present day family being pale skinned, don't forget each of your ggggg grandfathers makes up only 1/128th of your own ancestry.