The Athenaeum's collections resided briefly in Joy's buildings, Congress Street, but by the spring of 1807 were firmly established in Scollay's buildings, Tremont Street, near the present Government Center. The Athenaeum remained there until 1809, when the Trustees purchased the Rufus Amory House, adjacent to the King's Chapel Burial Ground at what was then the easternmost point of the Boston Common. In 1822 the growing collections were moved again, this time to the mansion in Pearl Street given to the Athenaeum by Trustee James Perkins. The first three floors of the present Beacon Street building, designed by Edward Clarke Cabot, were constructed between 1847 and 1849. The first floor was originally a sculpture gallery, the second housed the library's growing collection of books, and the third, with skylights, served as a painting gallery. The building was completely renovated in 1913-1914, at which time the fourth and fifth floors were added and the entire structure fireproofed. Architect Henry Forbes Bigelow designed these improvements.
The Athenaeum's five galleried floors overlook the peaceful Granary Burying Ground, and as Gamaliel Bradford wrote, "it is safe to say that [no library] anywhere has more an atmosphere of its own, that none is more conducive to intellectual aspiration and spiritual peace" (The Quick and the Dead, 1931). The building was declared a National Historic Landmark in 1966.