The details of Spivak's birth are unclear. Some sources place it in Ukraine in 1907, and that his family emigrated to settle in New Haven, Connecticut while he was a child. Others place his birth in New Haven two years earlier, in 1905. He learned to play trumpet when he was ten years old and played in his high school band, going on to work with local groups before joining Johnny Cavallaro's orchestra.
He played with Paul Specht's band for most of 1924 to 1930, then spent time with Ben Pollack (1931–1934), the brothers Tommy and Jimmy Dorsey (1934–1935), and Ray Noble (1935–1936). He played on "Solo Hop" in 1935 by Glenn Miller and the Glenn Miller Orchestra. He spent 1936 and 1937 mostly working as a studio musician with Gus Arnheim, Glenn Miller, Raymond Scott's radio orchestra, and others, followed by periods with Bob Crosby (1938), Tommy Dorsey (1938–1939), and Jack Teagarden (1939).
When the Spivak orchestra broke up, he went to live in Florida, where he continued to lead a band until illness led to his temporary retirement in 1963. On his recovery, he continued to lead large and small bands, first in Las Vegas, then in South Carolina. In Greenville, South Carolina in 1967, he led a small group featuring his wife as vocalist. She died in 1971 after a long illness with cancer.
During this time, Spivak was also resident band leader for a restaurant-nightclub, "Ye Olde Fireplace", in Greenville, South Carolina. He played trumpet in the dance band that included a drummer, saxophonist, bass player and pianist. The band played standards from the big band era, but also took requests from the audience.
On May 29, 1975, two gunmen entered the restaurant-nightclub after closing for an apparent hold-up. The suspects ordered everyone on the ground. One suspect encountered two band members in the band room, including long-time Spivak saxophonist Angelo "Charlie" Russo were talking. The gunman shot Russo in the chest, killing him.
Spivak continued to play and record until his death.