In 1922, Freeman and some friends from high school formed the Austin High School Gang. Freeman played the C melody saxophone with band members such as Jimmy McPartland and Frank Teschemacher before switching to tenor saxophone two years later. The band was influenced by the New Orleans Rhythm Kings and Louis Armstrong. While Armstrong was in King Oliver's Creole Jazz Band, Freeman attended performances at Lincoln Gardens with McPartland. They were nicknamed "Alligators".
In 1927, he moved to New York City, where he worked as a session musician and band member with Red Nichols, Roger Wolfe Kahn, Ben Pollack, and Joe Venuti. One of his most notable performances was a solo on Eddie Condon's 1933 recording, The Eel, which became Freeman's nickname for his long snake-like improvisations. Freeman played with Tommy Dorsey's Orchestra (1936–1938) and Benny Goodman's band in 1938 before forming the Summa Cum Laude Orchestra (1939–1940). Freeman joined the U.S. Army during World War II and headed a U.S. Army band in the Aleutian Islands.
After the war, Freeman returned to New York and led his own groups. He also worked with Buck Clayton, Ruby Braff, Vic Dickenson, and Jo Jones. In 1960, he wrote the book and lyrics for the Broadway musical Beg, Borrow or Steal, which included the ballad "Zen Is When", later recorded by the Dave Brubeck Quartet on Jazz Impressions of Japan (1964). He was a member of the World's Greatest Jazz Band in 1969 and 1970. In 1974, he moved to England and continued to record and perform. Freeman spent some time on the Isle of Man and was a guest of Manx musician Jim Caine. After returning to Chicago in 1980, he continued to work into his eighties.
He wrote two memoirs (You Don't Look Like a Musician (1974) and If You Know of a Better Life, Please Tell Me (1976)) and an autobiography (Crazeology) with Robert Wolf. In 1992, Freeman was inducted into the Big Band and Jazz Hall of Fame.
Freeman died March 15, 1991, at the Warren Barr Pavilion, a nursing home in his Chicago. He was 84 years old. His death came days after the March 13 death of Austin High School Gang member Jimmy McPartland.