Drummond Castle was built on a rocky outcrop by John, 1st Lord Drummond around 1490. The 2nd Earl, a Privy Councillor to James VI and Charles I, succeeded in 1612 and is credited with transforming both the gardens and the castle between 1630-1636. The keep still stands but the rest of the castle was restored and largely remodelled by the 1st Earl of Ancaster in 1890.
Early records mention Lord Drummond sending cherries to James IV in 1508, when the monarch was hunting in nearby Glen Artney Forest. The 2nd Earl must have owned a significant garden and the presence of the sundial (installed in 1630) goes some way to confirmong this.