It was quite common for married soldiers' wives and families to accompany them when their husbands/fathers were posted elsewhere.
In the 1851 census there are 183 people in 'Greenlaw Military Prison', parish of Glencross/Glencorse.
Of these 31 are described as 'Wife', 27 as 'Son', and 36 as 'Dau'. Go to
https://freecen1.freecen.org.uk/cgi/search.pl and search for Skiffington Briston (!) who was Governor at the time.
According to Historic Environment Scotland
"The clock tower [of Glencorse Barracks] likely dates from soon after 1803-1804, when Greenlaw House was given over for conversion to a prison for French prisoners of war. Alternatively, it may date from the wider development of the site as a military prison after 1813. The tower was previously encircled by a ground level timber lean-to, possibly containing stores. An equivalent, but taller, octagonal structure exists at HMP Perth, designed by Robert Reid and similarly used for prisoners of war. Glencorse prison was complete by 1813 and cost £100,000 (Groome.) It could accommodate 6000 prisoners and a plan shows observation walkways and prison blocks radiating from a principal terrace. This form may have survived the conversion to the general military prison for Scotland in 1845, but it was demolished either by or during the conversion to the central brigade depot for southeast Scotland in 1875-1877. Greenlaw House was also demolished, though the cellars may survive in the Officers' Mess block to the southeast. Glencorse Barracks remains in use by the military."