Gunzenhauser assisted Igor Markevitch in Monte Carlo and Leopold Stokowski in New York before becoming executive and artistic director of the Wilmington (Delaware) Music School in 1974. Five years later, he was appointed conductor and music director of the Delaware Symphony Orchestra and of the Lancaster Symphony in Pennsylvania. Gunzenhauser left the Delaware Symphony at the end of the 2001-2002 season but retained his Lancaster position even after he was named artistic advisor and principal conductor of the Bogota (Colombia) Philharmonic in 2004.
The State of Delaware appointed Gunzenhauser to be its first cultural ambassador in 1990. In December 1999, Gunzenhauser received the Order of the First State, the highest accolade awarded by the Delaware state government. He continues to live in Wilmington. Naxos and Marco Polo have sold more than two million copies of his recordings in repertory including Mozart, Vivaldi, Beethoven, Saint-Saëns, Brahms, Bruch, Orff, many Romantic Russians, Copland, Dvorák, and Chinese composer Chen Gang.