I was able to figure it out using atDNA. My grandad visited England and came home with a story that his dad was a Stuart-Dunn and was adopted by the Dunns but within the family. We have no idea where this came from but we suspect his much older sister old him. She also happened to name her only son Stuart and I don't think it's a coincidence.
The DNA lead me to his biological mother, who was a milliner, and then I was able to find his birth certificate with a father's name, last name Stuart. I traced the father's family to India and Northern Ireland. It's likely that this Stuart line in NI were given lands during the Ulster Plantation and they went over from Scotland.
I don't know how my grandad George ended up in Wollescote with the Dunns. His biological parents were basically playing house but he was married to another person. He seemed to do that a lot. I think it's likely that he took off or the mother realized he would never marry her so she took off. George's adoptive mother, Betsy Brooks, had a cousin who was a milliner and dressmaker. I wonder if that's where the connection was between the to families.