Nell, thanks for the link. In the 1841 and 1851 censuses Joseph's birthplace was Newcastle on Tyne. In 1861,71 and 81 it was Edinburgh.
I have found his christening at Tynemouth and the marriage of his parents, John Kay and Margaret Watson in Newcastle, so I know he was not born in Edinburgh!

Would he have been able to go in the workhouse in Edinburgh if he had not been born there?
His wife Ann [Bremner]
was born in Edinburgh.
She died in 1899 at Craigleith.
Kooky