There are Spanish names in Oban, Argyll and stories of the survivors of armada wrecks at places including Greenpoint near Poolewe in Wester Ross.
There are more definite historical accounts of armada survivors journeying across ireland, pursued by British noblemen, and making their way back to scotland at Kintyre, Argyll.
Spain and Portugal were allies of the Jacobites; Spain's wealth in particular, was a target of British (i.e. English-sponsored) Privateers in the caribbean and Americas.
Portugal is one of Britain's oldest allies, and there are still old British families in privileged positions in mainland Portugal and Madeira.
Then there's the trade links - Any of Scotland's ports would have had links with Spain and Portugal especially during the 19th and 20th Centuries.
There are some very interesting cases of Maternal DNA from exotic sources amongst people who thought themselves "pure" Scots - One in particular caught my imagination - A woman from the Lothians who has a maternal line going back to the South Pacific islands!
I'm rambling, as usual, but although I agree that, in the absence of extraordinary evidence of a specific connection, it is far more likely that swarthy looks are just part of the inheritance of some very British families, the converse may also be true - From the Peninsula Wars, via Captain Kidd, (from Greenock?), and merchant seamen of 19th & 20th Cs., there have been increasing opportunities for peely wally British men to acquire exotic brides and bring them home to Blighty, following the macho, patrilocal custom of the species.
T.