Isn't the rectory the building on the right in the original etching? (Although I'd describe the windows as "rounded" rather than "round" as Nat says.) In the 1960 photo there's a more modern, taller building in its place.
Although it's interesting that Nat confirms the demolition as seven years later than the official record has it, which shows how diaries can be useful for providing historical info as well as just social background.