Between 1836 and 1837 the hilltop was the site of Melbourne's first colonial burial ground. Then from 1840 to 1857, it became the site of a flagstaff which was part of a signalling system between Melbourne and ships sailing into the port at Williamstown. The site was then also used as a meteorological and magnetic observatory between 1857 and 1862. It was also the place from which news of Victoria's separation from NSW was signalled to Melbourne residents through the lighting of a great bonfire in 1850.
Established as a public garden in 1862, Flagstaff Gardens was designed in the gardenesque style in 1865 by Assistant Commissioner for Crown Lands and Survey, Clement Hodgkinson.
Along with the popular lawns, the gardens feature eucalypts, Moreton Bay figs, avenues of elm trees, shrub beds and a high vantage point. Facilities also include tennis courts, a playground, a public toilet, barbecues and a bowling green.