Construction of the house began in 1888 and was completed in 1895; the Architect-Builder was Charles A. Johnson.The exterior of the building is made of sandstone and limestone quarried locally and hand-chiseled. Native red oak is used throughout the interior of the building with hand-carved motifs adorning the windows and doors.

In 1885 Rufus Ayers served as Virginia's Attorney General. He and other gentlemen such as John Imboden, Charles Sears, George Carter, and John Taggart felt that Big Stone Gap could become the "Pittsburgh of the South" because of its iron ore and coal deposits. Rufus was instrumental in helping develop the coal and iron ore industry in Southwest Virginia and bringing the railroads to this area. Big Stone Gap, however, did not become the next Pittsburgh due to the economic depression.

The house was purchased by C. Bascom Slemp in 1929. Slemp, a native of Lee County, served many years in Congress and later became the private secretary to President Calvin Coolidge. C. Bascom and his sister, Janie Slemp Newman, had a love for Southwest Virginia, its people, history and rich culture. They collected artifacts depicting life of the area, which were originally displayed in the Janie Slemp Newman Museum. Before C. Bascom's death in 1943, he established The Slemp Foundation. It was his wish that the state acquire the Ayers' home for a museum and that the Janie Slemp Newman collection be given to the state for their museum.

In 1946, the Commonwealth of Virginia acquired the Ayers' home and the Slemp Foundation donated the collection. The Southwest Virginia Museum was officially dedicated by the state on May 30, 1948. The museum is managed by the Department of Conservation and Recreation's development of Southwest Virginia and the lives of the men and women who settled in and around the area. The exhibits depict the early "boom and bust" era of the late 1800's. Life at the turn of the century can be seen by such items as mail-order catalogs, photographs, and radios. Artifacts from the early settlers who developed the area in the late 1700's are also on display. Also in the collection are rare pieces that C. Bascom and Janie acquired during their travels. The collection includes a set of Disraeli china which Queen Victoria had made for her Prime Minister Benjamin Disraeli. Also included are fine French paintings, a cassone, which is an Italian hope chest, and antiquities from the Orient.