Yes, the best test would be a YDNA test of a direct male descendant of Charles Butler and one of Robert Hastings Smith (and hope that each man's father actually
was his father).

That would probably definitely perhaps give you the answers you want.
However ... approaching a relative of Charles Butler,
if you can find one, might need to be tackled carefully as they may know nothing of Charles, and if they do they may not believe he fathered an illegitimate child (some people look at their ancestors through rose coloured glasses). You may need to offer to pay for their test if they are reluctant to take one. Prepare yourself for them to refuse, but perhaps another descendant would be willing to help you.
Good luck and please let us know how you get on.