Have you contacted the library in Galway?
Is this right- your connection is not to Connemara but as an Audley you are curious about your surname occurring in that discrete area?
I have no idea and am not an Irish history scholar. However, theoretically, the name may have begun there with the arrival of just one person.

You question people 'emigrating' to remote islands but the flow of human traffic between England and Ireland was happening for a very long time - either willingly or unwillingly.
Searching for any historical references, there are references to the O Cadhlas but that wouldn't be pronounced with the 'd'' sound - I am trying to see a corruption of 'Cadhla' to 'Adley' but the name is recorded as O Kealey and perhaps Kaley/Kiley.
I hope you get an answer- it's interesting. One of my family names is Malley. Some people have O'Malley in the family, others Malley. The Irish is Maille and other members call themselves Malia - with a long 'a'. This is from a very small area where all are closely related.