In 1791 land here belonging to the Temple West family was leased to a Mr Hedger and houses were built. The garden was laid out by 1799. In 1812, the Admiralty installed a tower on no. 36 West Square for a shutter telegraph that conveyed messages between Whitehall and naval establishments in Kent during the Napoleonic wars. Senior staff from the Bethlem Royal Hospital (see above) were housed in the square in the early 1800s. By the end of the 19th century the garden was at risk from building development and a campaign was mounted to preserve it as an open space. The freehold was purchased for £3,500 in 1909 by the LCC and Borough of Southwark and the enlarged and restored garden was opened to the public in 1910.
After WWII there was a proposal to demolish the surrounding buildings and add the area to Geraldine Mary Harmsworth Park. The Civic Amenities Act prevented this and the square was designated a conservation area. After WWII prefabs were built in the square and the terraces in the north-west corner were demolished, although Charlotte Sharman School built in 1884-5 remains on the north-west side. In the garden are old mulberry trees, rose gardens and a tree planted to celebrate the square's bicentenary.