Well, the Infirmary (which was part of the Greenwich Work House) was on the corner of Vanburgh Hill at the Greenwich end of the hill, so it could have been that. The whole site later became St. Alfege's Hospital, which in its turn became the Greenwich District Hospital (now demolished). Try sending an email to the Greenwich Heritage Centre (you'll find it on Google) - they may be able to confirm.