The PCC wills can be ordered and received immediately from The National Archives electronically. To check whether Thomas Kempster left a will I would check with Essex Record Office (PCC Administrations have not been indexed electronically). Jemima died once the state (and not the church) had begun to probate all wills so if she left a will hers will be held by Her Majesty's Court Service - Probate service.
Both Thomas' will and Jemima's if they left one are less likely to give clues to Thomas' origins but you never know - obviously the earlier the will the more likely it will be to give some information which might help.
Another connection through records of the surnames Depree and Kempster to Bouverie Street
Wiltshire and Swindon Record Office
EARL OF RADNOR
Catalogue Ref. 490
Title deeds of individual places
London: Whitefriars property - Fleet Street, Bouverie Street, etc.
FILE [no title] - ref. 490/574 - date: 1812-1928
Parties: Jacob second earl of Radnor, Christie, Jacob fourth earl of Radnor, Symmons, William fifth earl of Radnor, Kempster, Depree, Gray, Wells, Thomson, Mackay, Jacob sixth earl of Radnor, [...] Oppenheimer, Apperly; Pictorial Newspaper Company (1910) Ltd., George first baron Riddall.
FILE [no title] - ref. 490/584 - date: 1816-1925
Parties: Jacob fourth earl of Radnor, Rogers, William fifth earl of Radnor, Reid's Brewery Co., Jacob sixth earl of Radnor, Watson, Room, Vale, Skelton, Jacob second earl of Radnor, Moyes, Smith, Ingle, Gale, Baker, Warde, Oldfield, Gadsby, Payne, Kempster, Cooke, Bradford, Arrowsmith, Willett, Blanford, Depree, Pullan, Beetham, Woodward, William third earl of Radnor, Churchill, Hedge, Garlick, Pechey.
The London Magazine 1821
Bankrupts
Kempster T Bouverie St, Fleet St carpenter [Templer 12 John St Minories T
(according to The Times a Thomas Kempster builder went bankrupt in 1844 - properties in Blackman Street, Borough, in Kent Street Southwark, in Cole Street, Dover Road and Vine Street, Minories)
The Depree connection to Buckinghamshire seems to predate by some years 'a moving out of London for merely retirement purposes'.
22nd October 1817
The Times
On Saturday 11th inst Mr G Griffiths, jeweller of Cursitor street to Catherine Jane, daughter of C T Depree, Esq of Foundling- terrace and Mecklenburg-hall High Wycombe Bucks
The National Archives has indexed quite a few Chancery documents which might concern your Deprees (difficult to tell without looking at the actual documents) plus also records such as
ED 49/357 Langley Marish: The Church House and Depree's Educational Foundation 1905-1906
WO 97/1209/262 CHARLES THOMAS DEPREE Born HOLBORN, London and Middlesex Served in Madras Artillery Discharged aged [Not Known] 1847-1850
Doesn't answer your initial question about Thomas Kempster's origins but are probably of interest in their own right. It might be worth contacting Buckinghamshire Record Office to see whether they hold anything further on the Deprees including any wills proved locally.
Regards
Valda