The only Bartlett found on either census for Penshurst was Hannah, who I assume was Reubens daughter:
ref HO107 1613c folio 724: Salmons Farm Penshurst (1851)
Hannah Bartlett serv un age 13 house servant born Chiddingstone
in the employ of John and Mary Saxby
It would appear that the younger Bartletts all stayed put, but the older ones had already left the area before 1841. Although the quality of both census returns is questionable, I'm pretty certain that I would have spotted more of them, had they been there.
Hope this helps with the research,
xx Carol.