The gardens were established in the late 1840s as part of the Norland Estate. A speculative developer, Charles Richardson, built the houses around St James's Gardens and set up the gardens for the benefit of the residents who were, and still are, required to maintain them. Richardson gifted the site for St James's Church to the Church Commissioners on the basis that the houses would be more saleable if they had access to their own adjacent church. The church, designed by architect Lewis Vulliamy, was built between 1844 and 1855. The gardens are laid out in an informal woodland style, dominated by the fine chestnut and lime trees which date back well into the 19th century.