The men wore basically dark knee length trousers, socks and shoes (if they were lucky), a whiteish collarless shirts and a waistcoat and jacket. The sketches by Hogarth gives a very good idea.
As to washing, many Victorian houses had a "copper" in a shed in the yard. This was basically a big pan over an open fire that clothes were boiled in. Most people did not wash their clothes, many only did so a few times a year at most.
But they did not think they smelt as much as you might think! They were all the same, so no one noticed. It was just a natural background. But the smell of modern soap and chemical fresheners etc might be considered a smell by their standards!
The Hogarth link also implies what their world was like.
Rod