The column for born outside England (if you are looking at the English 1841 census) refers to where they were born, not their nationality. The only choices were S for Scotland, I for Ireland, W for Wales or F for foreign parts.
So for people born in the Isle of Man, only F would fit as the Isle of Man was not a part of England, Scotland, Wales or Ireland. And I have in fact seen someone born in the Isle of Man referred to as born in Foreign parts, on a 1841 Census of Cumberland. He was a MITCHINSON.