You are probably best continuing your research to establish some facts.
We are talking a long time ago, so I doubt anyone in Hove will know your grandmother. If anyone knew of the Denyer families, I doubt they would have known them well enough, nor the information been passed down the generations accurately enough, for them to distinguish between the two Alfred Denyers we have found.
Just to summarize.
Alfred Jupp Denyer married Mary Jane Smith 1897. She already had a child Frederick Smith before this marriage.
In 1901 she , with 7 yr old Frederick, is visiting the Andrews family, whilst her husband is in Cold Waltham, he is a groom/horse keeper. She is born Cold Waltham, son bn London.
By 1911, Frederick has been sent away. Alfred and Mary Jane in Horsham * see next post - no they aren't in Horsham!.
Meanwhile -
An Alfred Denyer married Mary Walder in 1895. They had children Mary Victoria Walder, Nathan Thomas Alfred and Edward Charles, and a couple of others that died.
In 1901 they are in Portslade
In 1911 at Wish Cottage.
Assuming there is a connection between the families, as you say Frederick had a photo of his mum Mary outside Wish Cottage with Victoria, it must presumably be via the Denyer link.
i.e the Alfred Jupp that Mary married must be linked to the Alfred that lived in Wish Cottage.
Are you sure the photo was of his Mum, and not of the Mary (nee Walder) who lived there?
Presmuably not brothers, both being Alfreds. So as someone has already said, you will need to delve deeper.