As a boy he studied piano and violin; most of his career was spent on cornet. Stewart dropped out of high school to become a member of the Ragtime Clowns led by Ollie Blackwell. He was with the Musical Spillers led by Willie Lewis in the early 1920s, then with Elmer Snowden, Horace Henderson, Fletcher Henderson, Fess Williams, and McKinney's Cotton Pickers. In 1933 he led a big band at the Empire Ballroom in New York City. Beginning in 1934, he spent eleven years with the Duke Ellington band. Stewart co-wrote "Boy Meets Horn" and "Morning Glory" and supervised recording sessions by members of the Ellington band. He left Ellington to lead "little swing bands that were a perfect setting for his solo playing." He toured in Europe and Australia with Jazz at the Philharmonic from 1947 to 1951.
Beginning in the early 1950s, he worked in radio and television and wrote jazz criticism for the Los Angeles Times and the magazines Playboy and DownBeat. The book Jazz Masters of the Thirties is a selection of his criticism. He lived in upstate New York after purchasing a one hundred year old farmhouse. He hosted a jazz radio program in Troy, New York, and owned a small restaurant for a short time near a drag racing track in Vermont. While living in France, he attended the Le Cordon Bleu school of cooking and dedicated his life to becoming a fine cook. Stewart moved to Los Angeles, California, to be near his children. His son Paul Albert Hardy lived in New York City. While in Los Angeles he reunited with musicians from the Ellington band and played jam sessions in clubs. He was a studio musician for The Steve Allen Show and with George Cole he hosted two radio shows: Dixieland Doings and Things Aint What They Used to Be.