Thank you for muddying the waters because we do wish for a clear picture of their movements and where they lived.  Also... I need to ensure I am following the right Paterson line.    Which  brings me back to where I read about the cottage... and it clearly states Dumbarton road???
I am pleased you are interested in this puzzle.  Any guidance would be wonderful Cheers.
1881 census says
"The cottage Clyde Back, Dumbarton Road, Old Kilpatrick, West or Old Kilpatrick, Dunbartonshire, Scotland".
First name(s)   Robert
Last name   Paterson
Relationship   Head
Marital status   Married
Gender   Male
Age   53
Birth year   1828
Occupation   Farm 100 Arable Acres 3 Men & 2 Women
Birth town   Stone House
Birth town as transcribed   STONE HOUSE
Birth county   Lanarkshire
Birth county as transcribed   LANARKSHIRE
Birth place   Scotland
House name   The Cottage Clyde Bank
Street   Dumbarton Road
Town   -
Parish   Old Kilpatrick
County   Dunbartonshire
Country   Scotland
Registration district   West or Old Kilpatrick
Archive reference   RG11
Robert   Paterson   Head   Married   Male   53   1828   Farm 100 Arable Acres 3 Men & 2 Women   Stone House, Lanarkshire, Scotland   Transcription
Ann   Paterson   Wife   Married   Female   43   1838   -   Renfrewshire, Scotland   Transcription
Jane   Paterson   Daughter   Unmarried   Female   16   1865   School   Dunbartonshire, Scotland   Transcription
Samuel   Paterson   Son   -   Male   12   1869   School   Dunbartonshire, Scotland   Transcription
James   Paterson   Son   -   Male   9   1872   School   Dunbartonshire, Scotland   Transcription
John   Paterson   Son   -   Male   6   1875   School   Dunbartonshire, Scotland   Transcription
Lillias   Paterson   -   -   Female   1   1880   -   Dunbartonshire, Scotland   Transcription
Annie   Smillie   Servant   Unmarried   Female   21   1860   Dairy Servant   Port Glasgow, Renfrewshire, Scotland   Transcription
Isebella   Lamont   Servant   Unmarried   Female   24   1857   Dairy Maid   Bunessan, Argyllshire, Scotland   Transcription
Thomas   Stewart   Servant   Unmarried   Male   34   1847   Farm Servant   Dunbartonshire, Scotland   Transcription
Robt   McGibon   Servant   Unmarried   Male   25   1856   Farm Servant   Renfrewshire, Scotland   Transcription
Robert   Paterson   Son   Unmarried   Male   14   1867   Schollar   Dunbartonshire, Scotland   Transcription