Phillips made the first team as a freshman on the Concord football team and started every game. In 1958, he led Concord in rushing and scoring and was second team all-conference.

In 1959, Joe Ed was second in the conference in rushing and scoring, was first team all-conference and Honorable Mention All-NAIA. In 1960, he again led Concord in rushing and scoring, was First Team All-WVIAC and Honorable Mention All-NAIA.