Hi Dee, 
I have found another James Draper on the 1851 with a son Joseph and these appear to fit the criteria of your original posting. 
HO107/1707  F.187.  P.5.  Southwell Greeen, Rushden
Living with Benjamin and Mary Cannon and family, farmers of 262 acres of land employing 14 labourers
James Draper  Servant  Widower  40  Ag Lab  Rushden
Joseph Draper Servant  U  21  Ag Lab  Rushden. 
Perhaps  this will affect your line of enquiry and I will wait for your reply. 
Regards   ...   Peter