Thank you for the 1841 census info. It looks very possible.
I'm interested to know why you mentioned Alexander's eldest "daughter" in quotes, and if you know something, or could help..
The mother of Eliza(beth) - and the other 2 children, John and Alexander from the 1851 Johnstone census - appears indeed to be Martha Wylie, and not Agnes Stewart.
Alexander Brown, moulder, married Agnes Stewart in Johnstone in 1846 (there are 2 records of bans anyway on 23 Aug 1845 and 7 Feb 1846).
Moreover, there is also a record of Alexander Brown, moulder, having married Martha Wylie in Paisley on 21 Aug 1845.
Alexander Brown and Agnes nee Stewart moved to Northern Ireland in the 1850's and had other children, including my gg grandmother. There are several Wylies appearing on subsequent records there (one as a lodger on the 1911 census, and one as a marriage witness). The plot thickens.
I can't find any further info on what happened to Martha Wylie/Brown.
Two Alexander Browns?

Thank you again.