Hi islander,
from an 1854 map there was and still is North High Street then Bridge Street then again onto the High street the latter is where the Town Hall is located so I would say it was Main High Street.
Where the Harbour is this is called New Street, Fisherrow eventually working its way to Bridge Street
Cheers.
Archie.