The garden was originally conceived in 1930 after Cason J. Callaway discovered a rare azalea growing in the area. Callaway Gardens opened on May 21, 1952 as the Ida Cason Gardens, with a number of lakes, a golf course, and scenic drives. The gardens were named for the mother of founder Cason J. Callaway. Robin Lake Beach and the Overlook Azalea Garden opened the following year in 1953. In 1955, The gardens were renamed Ida Cason Callaway Gardens. The Masters Water-ski Tournament, now an annual event, held its first competition in 1959.
On April 12, 1961, founder Cason J. Callaway died and was succeeded as Chairman of the Board by his wife, co-founder Virginia Hand Callaway.
The gardens have experienced numerous expansions following Cason Callaway's death. The Cason J. Callaway Memorial Forest opened in 1972, and was designated a National Natural Landmark by the United States Department of the Interior. The John A. Sibley Horticultural Centre opened in 1984. Mr. Cason's Vegetable Garden was the location for years of TV shows about growing vegetable gardens, most notably the southern edition of The Victory Garden. The annual Steeplechase at Callaway Gardens ran its first race in 1985. The Cecil B. Day Butterfly Centre opened on September 25, 1988. "Fantasy in Lights", a Christmas light display, debuted in 1992. In 1999, the Azalea Bowl opened as well as the premiere of the Sky High Hot Air Balloon Festival. In 2000, the Virginia Hand Callaway Discovery Centre was opened
