As long as you descend in the male line, then you have the right.
If this is England/Wales, then that's the end of it.
If Scotland, then the Law still applies, and you can be prosecuted for using Arms to which you are not entitled.
Quoting from The College of Arms website:
For any person to have a right to a coat of arms they must either have had it granted to them or be descended in the legitimate male line from a person to whom arms were granted or confirmed in the past.
