I've found "Little London City" on an old large scale Ordnance Survey map, about a mile north of the town centre. The words appear in the triangle formed by the roads now shown as Lee Green, Old Bank Road and Greenside Road, and while I can't be certain, I think they might refer to a collection of buildings just north of Lee Green, between that and what is now shown as Savile Place. I've no idea whether the same buildings are still there now.
I'm not sure why an area would be called Little London - it's certainly not the only one. Maybe it was once settled by a group of families from London; or in view of the typical Yorkshire disdain for all things southern, perhaps it was thought to be particularly crowded, noisy, dirty etc.
Arthur