Originally set aside as a reserve in 1848, the gardens were named after Sir Charles Augustus Fitzroy, Governor of NSW. The gardens were laid out in 1859 and were developed as a garden under the supervision of Deputy Surveyor-General Clement Hodgkinson, with the assistance of gardener James Sinclair.
The layout follows a classic Victorian era design, featuring pathways lined with magnificent elm trees, known to be some of the best grown in Victoria. There are a variety of flowers and ornamental shrubs and trees, which together with extensive lawns creates a diverse and layered landscape.
The watercourse that runs through the centre of the gardens is an ephemeral tributary of the Yarra River, Birrarung. The Scarred Tree in the south-east section of the gardens is associated with the pre-colonial use of the area by Aboriginal peoples and it is registered on the Aboriginal Heritage Register. The tree was a redgum, one of a number of remnant redgum found in the gardens.
The gardens contain several buildings and structures which are of architectural significance in their own right, including the Band Pavilion (1864), Sinclair's Cottage (1866) and the Rotunda (1873).
Significant twentieth century buildings include the plant manager’s cottage The Lodge (1927), the recently refurbished Spanish mission-style Conservatory (1930) and the Electricity Substation (1940).
The Fitzroy Gardens Visitor Centre opened in 2014 and provides tourism information and manages entry to Cooks' Cottage.