Rifkin first made an impact in 1965 when Elektra Records impresario Jac Holzman asked for help in realizing a crazy idea of his, an album of Beatles hits as restated in the musical style of Baroque composers. Witty and unique, the resulting Nonesuch album The Baroque Beatles Book swiftly became a best seller, featuring Rifkin's clever take on Beatles standards such as "Help!" and "Please Please Me," and it was widely imitated. In 1970, Rifkin began an extensive project of recording Scott Joplin's rags for Nonesuch that stretched into 1974 and resulted in three LP volumes. With the release of the hit movie The Sting featuring Joplin's music, theretofore mainly known to early jazz buffs, Rifkin's recordings of the piano originals were thrust into prominence and also became best sellers. So enormously popular were Rifkin's efforts on behalf of Joplin that for the rest of the 1970s his was a household name among classical music fans as well as a cross-section of popular music listeners. Rifkin ended the 1970s on a high note with Digital Ragtime, an album for EMI featuring the first digital recordings of ragtime piano music; it also proved very popular.
Despite the success Rifkin enjoyed as an interpreter of Joplin's piano rags, he did not lose sight of his strong interest and expertise in Baroque vocal music. In 1982, Rifkin recorded for his edition of Johann Sebastian Bach's B minor Mass for Nonesuch that proved a highly controversial sensation; it employs only one singer per part in the chorus rather than the sizeable choral forces that then proved the standard in modern performances of the work. Although Rifkin's proposal of this idea was contested in many quarters at the time, it has since been widely adapted among groups that perform Bach's music in period style. Rifkin has gone on to record several of Bach's cantatas, Haydn symphonies, and works of Antoine Busnois; in 2004 he published a scholarly monograph on the music of the Beatles.