On FindMyPast I typed, in general, the 1851 census year, and over 20'000 people lived in Oxford city in 1851, so I typed in a keyword such as "Devon" to see how many were in Oxford in 1851 and it was about 200, and about 1000 or so residents of Oxford in 1851 were born London/Middlesex, and around 1500 from Berkshire, about 400 from Bucks, 300 from Gloucestershire and inbetween 80 to 200 people from places like Devon, Norfolk, Essex, Kent, Suffolk etc, and only about 30 from Dorset and 30 from Sussex. A clever search engine way of finding out how many from another county were living in a certain English city in 1851. FindMyPast does have good keyword search features. In regards to James Smith, it is likely he was from perhaps Berks, Bucks, Gloucs or London but could have been in the smaller bracket of strays in Oxford who came from further afield.
I have an Inkpen surname in Oxford in the 1760s and the surname is a Dorset and Sussex name.
If I find another secret illegitimate child thanks to the 1921 census I think "Shall i tell them" to cousins in contact, or just let them see my updated tree over time, and they find out.