Sometimes it's just someone who fails to live up to the family's expectations. I was jokingly referred to as the "black sheep of the family" once by a cousin because all the others were in the "caring" professions, i.e. teaching, medicine etc, and I worked in law. Although in my case it was just a joke, it could be quite serious if, say, someone was born into a very talented musical family who all played instruments or sang, and that one person just wanted to live a nice, quiet, dull life, getting married and having children. Could be big disapproval from The Family!