Passmore has been involved in boxing one way or another for most of his life, beginning at the age of nine. He was a successful amateur boxer, winning several tournaments in North and South Carolina and has been a driving force in the sport since leaving the ring. He was a referee/judge for over 35 years, ran the Silver Gloves tournament in Gastonia for 13 years and was instrumental in the establishment of Golden Gloves tournaments in Kings Mountain and Lincolnton, helping to operate those events for years. Passmore has served as State and Regional Chairman of the United States Amateur Boxing Association (USABA) and served as National Chairman of the Junior Olympics for three years, and National Registration Chairman of amateur boxing for three years.
He and wife Sammie are the parents of four children. They also have six grandchildren.