A little thought scenario...
There is nothing to state he was left in the workhouse, only the birth which Elizabeth/Betsy would have used for birth/maternity care, as was the case for the poor outside of home delivery at that time.
As mentioned, it could be possible that Betsy sent money for his care. When infants it was called nurse child.
The info found by Ciderdrinker, reply #20, that father James was living Church Street, SWEENEY in 1821 and family at Church Street in 1841 and 1851 it is a very good possible that the Jones knew the family, or knew of them. (Family were also innkeepers/publicans)
Also as mentioned there is a good possible that he may have a Jones father that is connected to Joseph or Elizabeth. Hence the Jones surname. I would think to try prove/disprove this is Edwards bio family, would be through DNA, as that is the only truth teller nobody can escape from.
Just a few thoughts....