Charles A. Johnson was contracted by Rufus Ayers to build his home in Big Stone Gap. The beautiful Victorian stone mansion, that houses the Southwest Virginia Museum, is a legacy to the craftsmanship of Charles Johnson and others who assisted him. Charles A. Johnson was born Charles Gustus Johnson in 1865 in Sweden. He was educated at the University of Stolkholm and was fluent in languages. He immigrated from Stolkhom to America and came to Pennsylvania, where he was later naturalized. Charles A. Johnson was called Charlie, or sometimes referred to as Charlie "the swede" Johnson. Charlie came to Southwest Virginia with the railroad to build stone trestles for the train tracks. There he became a stone mason and contractor of some note.

In Big Stone Gap, he built the Ayers Home from sandstone and limestone cut from nearby Stone Mountain. While building the Ayers Mansion, Charlie met and fell in love with Rufus Ayers' housekeeper, Zella Missouri Payne. Zella was originally from Happy Valley, Tennessee, and according to family oral history, was related to Davy Crockett. Charlie and Zella lived in the Cadet section of Big Stone Gap. They had four children: Charles R. Johnson, Hattie Elizabeth Johnson, Mary Johnson, and Grace Johnson.

In addition to the Museum, Charlie Johnson built several other stone structures in the area. He built many of the stone walls in Wise County, including the wall and fountain at Glencoe Cemetary. He also built the stone structures at the Wise County Poor Farm. Today, these are the original stone buildings of the University of Virginia's College at Wise.

Charlie Johnson is fondly remembered for building the three stone buildings of the Big Stone Gap public school. Unfortunately, these magnificent buildings were torn down. The head stone and architect's stone were donated to the Museum and can be viewed on our Walk of the Grounds.

Charlie Johnson died in 1937. He is buried in Glencoe Cemetary, lovingly surrounded by the monuments to his craftsmanship.