"does that mean, in 10 generations i.e. on average, about 300 years, the likelihood of your ancestor in the male line being someone of a different surname is 50%?"
No, it doesn't.
For the purposes of argument, let's assume that 5% is an accurate estimate of the illegitimacy rate and that that rate has not changed over time. If you look at ten generations of direct ancestors in the male line, you have a sample size of ten paternity events of which 5% [on average] are likely to be illegitimate. In other words, in that specific line - the one bearing the surname - the likelihood of an illegitimate birth is low, less than one out of the ten.
Obviously, if you consider all of your direct ancestors over ten generations [not just the male line], the chances of an illegitimate birth are much higher. About 50 illegitimacies could be expected [0.05 x 1024 = 51.2].