Annie was John Lee's second wife, both wives died young due to illnesses, Annie was 25 & John was 38 when they married in 1881, maybe he moved on & found another partner but again with regard to the son John William who was just 2 years old when his mother died in 1884 he was born in Annie's family home in Blackburn, so my thinking is that Annie's family would have taken him in if his father decided to move on rather than him being in put in the workhouse in Blackburn.
Just my thoughts.