Have you checked if there are any witnesses to James marriage who could be related.
Have you looked in the records for St Ann Richmond for 'Youngs' who could be Mary's late husband (ie their death) or her children.
Is there a bastardy order for James.