Seems that folk were struggling with the term nearly 120 years ago, an article in the Taunton Courier and Western Advertiser of May 1899 included the following observation:
... Of a morning I have seen as many as a half-dozen persons washing their faces and arms with dew on a fine May morn, but I don't remember that it removed the freckles, or, as they were more commonly called, "summer-voys," a word I cannot understand or find in any dictionary ...