Rufus Ayers was born on May 20, 1849 in Bedford County. His family moved to Goodson (now Bristol) in 1855 where he attended Goodson Academy until it was closed at the start of the Civil War. At age 14, Rufus ran away and joined the Confederate Army. After the war, Ayers began to study law and, in 1875, became the Scott County Commonwealth Attorney. In 1876, he obtained a charter for a railroad from Bristol to Big Stone Gap. Ayers quickly became one of the state's foremost industrial leaders.

Ayers became involved with politics as a member of the Democratic State Committee of the Ninth Congressional District in 1883. The next year, he was Vice-President of the Virginia delegation to the Chicago Convention, where Grover Cleveland was nominated. In 1885, Ayers was elected Attorney General of Virginia. As Attorney General, Ayers went to jail as part of his fight on the state debt issue. He had his case heard before the U.S. Supreme Court, which ruled in his favor.