Beginning his career in 1945, Pappas got off to a bad start. Boxing with the YMCA League in Winston-Salem, the 11-year-old lost his first seven bouts. After that Pappas began to win, and of his next 130 fights, he lost just five. He won seven amateur championships, including the High Point Tri-State Golden Gloves, as a lightweight, in 1951. Pappas moved into the Open Division at the age of 15 because of a lack of novice competition. The move did not slow the youngster down. Once, his team fought a match against Mooresville Mills, and Pappas boxed the coach and won. He fought at the Carolinas Golden Gloves in Charlotte and was beaten by a much older fighter, 82nd Airborne Division Champion Gilbert Ruybalid. The paratrooper won by an eyelash, but the decision was roundly booed.
Pappas served in the U.S. Army from 1953-1956, where he won two service tournaments.
Pappas is married and he and his wife Carolyn have three daughters, seven grandchildren and one great-grandchild. Pappas retired as food service manager of Nash Health Care Systems in Rocky Mount, N.C. in 1997, having served 20 years in that capacity.