This seems to have got in a muddle.
Brunswick Street - this is where the Leamington Municipal cemetery opened in 1868 - and when it opened a section was set aside for Leamington All Saints church. There were not any burials in this cemetery before 1868
All Saints parish church, only had a very small graveyard next to the church. When this was full they used a piece of land in leam Terrace and when that was full they used a piece of land in new Street - after which the problem was solved as they got a section in the municipal cemetery in 1868.
So you could check All Saints to see if he was buried there.
As though the gent lived in Milverton, he could have been buried in his own parish church at Milverton or possibly in the parish of St Marks on the Rugby Road - these are two other option to investigate.
The municipal cemetery records are held by the district council
the parish church yard records are held by the record office in warwick.
The Leamington Courier is online at British Library - have you searched the index?