Ernie Freeman began playing in local Cleveland area nightclubs in 1953,  and also formed a classical music trio for local social functions with his father and his sister Evelyn. Around 1939, he and Evelyn formed a new band, The Evelyn Freeman Swing Band, with fellow teenagers from Cleveland Central High School. Evelyn played piano, while Ernie played saxophone and also began writing arrangements for the band. The band began a regular engagement at the Circle Ballroom in Cleveland, and broadcast shows for WHK radio station. In 1942, most of the band, apart from Evelyn, joined the US Navy together, and became the first all-black Navy Band, called "The Gobs Of Swing", with Ernie as its leader.

After leaving the Navy in 1945 Ernie entered the Cleveland Institute of Music and then moved to New York City. After a spell as arranger for Woody Herman he joined the Ernie Fields Orchestra, playing the piano. Other members of the band included saxophonists Earl Bostic and Plas Johnson, guitarist Rene Hall, and drummer Earl Palmer. In 1951 Freeman also began playing with the Billy Hadnott Sextet, but left in 1954 to form his own combo with Johnson, Palmer and guitarist Irving Ashby. In 1955 they released their first record, "No No Baby" on the Middle-Tone label. They also recorded with a vocal group, The Voices, who included Bobby Byrd and Earl Nelson of The Hollywood Flames (and later Bob and Earl).

Freeman played on numerous early rock and R&B sessions in Los Angeles in the 1950S, particularly on the Specialty, Modern and Aladdin labels, as well as for white artists such as Duane Eddy and Bobby Vee. He played piano on The Platters' "The Great Pretender" in 1955, and began releasing a number of instrumental records of his own. These included "Jivin' Around" (#5 on the R&B chart in 1956), and his version of Bill Justis' "Raunchy", his biggest solo success, which reached #4 on the pop chart and #1 on the R&B chart in 1957.