Kalvi is best known for its Medieval manor. It was first mentioned in 1485. The owners von Lodes had built there one of the grandiosest vassal fortresses in Estonia. Ca. 30 m wide trapezoid-shaped castellum type fortress was probably built in the beginning of the 15th century. It is also possible that the manor had existed already in 13th–14th centuries.
A new Early-Classical main building was erected on the eastern wing of the fortress by the von Essens. In 1910 it burned down and was replaced by a new luxurious eclectic main building nearby in 1913. The manor was owned by the von Stackelbergs until 1940. Nowadays a hotel and a restaurant operate in the building.
Russian military commander Ivan Essen (1759–1813) was born in Kalvi Manor. Soviet Estonian political leader Johannes Käbin (1905–1999) was born in Kalvi before immigrating to Russia
