Born in Spartanburg, S.C., Baker began his boxing career at age 12 under the tutelage of the late Henry “Pappy” Gault. He boasts a record of 27-4 as an amateur. He won several Golden Gloves championships in both states. Baker moved to Washington D.C. in 1973 and turned professional. He beat Leo Saenz in 1975 for the U.S. Jr. Middleweight title and two years later became the property of the Washington Redskins when Redskins owner Edward Benett Williams purchased his contract.
Baker won 50 pro fights and lost 19 with three draws. He racked up 32 knockouts. Retiring from the ring in 1984, Baker worked as a supervisor with the U.S. Senate Postal Service from 1985-2000. He is married to the former Dianne Brazelle of Amelia Island, FL.
Baker once fought six pro fights in 20 days, a record at the time. He also fought Maurice Hope for the World Super Welterweight title in 1979 but lost on cuts. Many observers said he was ahead on points at the time.