Jimmy Dorsey was born in Shenandoah, Pennsylvania, United States, the first son of Theresa Langton Dorsey and Thomas Francis Dorsey. His father, Thomas, was initially a coal miner, but would later become a music teacher and marching-band director. Both Jimmy and his younger brother, Tommy Dorsey, were musically active during their childhoods and by the age of seven, Jimmy was already playing with his father's band. He made his first public appearance at the age of 9 while playing trumpet with J. Carson McGee's King Trumpeters in New York in 1913. He switched to alto saxophone in 1915, and then learned clarinet. Jimmy Dorsey played on a clarinet outfitted with the Albert system of fingering, as opposed to the more common Boehm system used by most of his contemporaries including Benny Goodman and Artie Shaw.
During his early days as a musician, Jimmy Dorsey performed with various ensembles and artists, including the Scranton Sirens, The California Ramblers, Red Nichols, Jean Goldkette, Frankie Trumbauer, Ben Pollack, and Paul Whiteman. He played the clarinet solo on the iconic 1927 jazz standard "Singin' the Blues", with the Frankie Trumbauer Orchestra featuring Bix Beiderbecke (which would, a half-century later, be inducted into the Grammy Hall of Fame). After returning to the United States from his European tour, he worked briefly with Rudy Vallee, and with several other bandleaders; and likewise with his brother Tommy—including starting their famed eponymous band. He appeared on at least seventy-five radio broadcasts, many of them with his brother. He was a member of Nathaniel Shilkret's orchestra, on programs such as (starting in 1937) "The Music That Satisfies" (also known as the Chesterfield Quarter Hour). Glenn Miller arranged and played trombone on several early sessions that he and his brother Tommy did together for OKeh Records, including "The Spell of the Blues", "Let's Do It", and "My Kinda Love"—all with Bing Crosby on vocals.
Jimmy and Tommy Dorsey reunited on March 15, 1945, to record a V-Disc at Liederkranz Hall in New York City. Released in June 1945, V-Disc 451 featured "More Than You Know" backed with "Brotherly Jump". The songs featured the combined orchestras of Jimmy and Tommy Dorsey. In 1947, Jimmy signed with MGM Records and in the same year, the brothers would put aside their tensions to film The Fabulous Dorseys.