The first chapel on this spot was built by Anton Kobald in 1853 after recovering from a severe illness. The name Klein-Mariazell was used because the relief madonna on the altar resembles the statue of Our Lady of the basilica in Mariazell. As the story of the miraculous healing spread, many people started to make pilgrimages to this site. In 1856, the chapel received the licence for the celebration of holy masses on all weekdays. Soon, the chapel became to small to accomodate all pilgrims so that Kobald decided to build a new, bigger chapel. From 1883 to 1888 the old chapel remained inside the new construction. The new church was finished in 1890. Klein-Mariazell became a parish in 1926. The village belongs to the municipaliy of Eichkögl.
Another pilgrimage site named Klein-Mariazell is located in the municipality of Altenmarkt an der Triesting in Lower Austria.