I am treading a little on Carol's toes here

as she has worked hard to find that family in 1881, and also 1891. I have now got them in 1901 too (I expect Carol has too).
Here are the details:
1891 (found by Carol, not me)
Baildon
RG12 3530 109 17
George 46 Cotton Warp dresser
Rachel wife 40
Lottie 17 worsted weaver
Mary Ann 15
Ettie 13
Jesse (indexed Jane!) 10
Edward 7
George William 5
and in 1901
Rg13 4067 83 33
Oak Place Baildon
William 57 Worsted warp dresser
Rachel 50
Charlotte 27 Worsted weaver
Mary A 25 "
Ettie 23 "
Jesse 20 Drapers assistant
Edward 17 Apprentice to AMrket gardener
George William 15 wool sorting.
I think it is highly likely that this is your Jesse's family.
Although they seemd to start in cotton, I always thought "worsted" was wool , though I may be wrong. Anyway, wool is gradually creeping in as little George sorts wool.
There is, alas, no Robert. But I do wonder if robert's name has been misremembered over theyears? could th emystery brother that Auntie Mabel visited possibly have actually been George William? or Edward?
( Could George William have been known as Robert to differentiate from father William and older brother George?)
Looking at the 1901, we know YOUR Jesse marries in 1907 - so will be interesting to see how old he was then and who his father was, and his occupation. That will help confirm if this one in 1881/91/01 is the right Jesse.
If it is then sorry, there is not a brother called Robert unless born after 1901 - but mum is 50 so rather unlikely.