Hi, 
Was Caswell cottage maybe run as some sort of girls home or school? I just found the 1901 census entry and it reads:
RG13/5083 f87 p27
Caswell Cottage
G H Dilwyn Llewellyn    b.1862  female    bn.Coln St Aldwyn, Gloucestershire, England    Head    Oystermouth    Glamorgan    
occupation Matron of home
Beatrice M Daniel    b. 1868    bn Brockley, London, England    Mext?    Oystermouth    Glamorgan under matron
Then some servants..
Then some young girls aged 8-13 listed mostly as boarders occ.scholar...one is a visitor. (sorry for not listing them I'm just a bit busy today)
The girls seem to come from all over and not to have any obvious relationsip with the Llewelyns.
Hope this is some help,
Emily
The GH Dilwyn Llewllyn seems to be Gwendoline, a daughter of John Talbot Dilwyn Llewelyn.