Unfortunately it is not easy to track the children born to slave owners in the Caribbean. On the one hand slave owners tend to own the children born out of wedlock, they also determine the status of the mother, that is whether to have the children legally recognised or not. However, I have found when tracing children accepted as either natural or reputed child born to slave owners - they tend not to appear on the slave returns. This can make it difficult to identify the mother.
Regards the term brown man - it is synonymous with other terms such as mustee, quadroon and as you said mulatto. Crude techniques were employed when classifying the complexion of an individual at the time of baptism.
I have the baptismal registers for children of John Alexander Sawers[Sawyers] snr. Do you require them?I can email them to you if you wish. Although you probably have them already. Another theory - found a John Sawers, baptised 1798 in parish of Hanover aged about 8 years old. No mother named. Listed as a quadroon child that makes him a strong possibility to be John Alexander Snr - why - in Jamaica and many countries many individuals were given one first name, as the individual approach adulthood, tends to take on other names. Found another John Sawers, quadroon baptised in St James in 1785, again no mother named. I realise that these are conjectures. Quadroon children were usually born to a white man and a mulatto woman.