You know that James Morrison was alive in 1861, and dead by 1871.
He said in 1851 that he was 42, and in 1861 that he was 49. so he was born some time between about 1810 and 1820.
There are five deaths of James Morrisons, born 1815 plus or minus 5 years, in Glasgow between 1861 and 1871. See screenshot.
The census index at SP shows that in 1851, 1861 and 1871 the family was in Central District 644/1. So the chances are that your James is one of the three who died in Central.
Checking the mothers' maiden surnames against the old parish registers, there is a baptism of James Morrison to Archibald Morrison and Anne MacCallum in Glassary in 1810.
I couldn't resist having a look and bingo! James Morrison, married to Mary Morrison, parents Archibald Morrison, weaver, and Agnes McCallum, died at 12 Weaver Street, Glasgow on 15 July 1868. (The names Agnes and Anne can be interchangeable.)
This couple were married in Glassary in 1803 and had 8 children baptised in Glassary between 1802 and 1827. See screenshot.
In the 1841 census, in Glassary, are a family consisting of Archd, 51, weaver, Ann, 60 and Ann, 19. These look rather like your Archibald and Ann(e)/Agnes. See screenshot.
In 1851 Ann is a widow, aged 60, with daughter Ann, 24 and two young grandchildren. They're not there in 1861.
So that gets you back another generation, and provides lots of scope for exploring lateral lines.