Since you have no specifics re. the marriage you can't really know that it did take place in Gorleston. Mutford regn. district covered numerous parishes and it could have taken place at any of them.
Also, as pointed out already, there were 3 James Smith's baptised in 1817 - not knowing definitely where the marriage took place your only option seems to be to obtain a copy of the marriage certificate to at least establish his fathers name and occupation to establish which James he was. If he was James George Smith bp.30/9/1817, son of James and Charlotte, he is their only child baptised in Gt. Yarmouth.
There are some online trees for James and Lydia but none of the contributors have got this marriage certificate - without it, it's just a case of if's, but's and maybe's. With a name like Smith you need to be certain.
Annette