Duncan and Curtis Williams, former high school classmates, formed the group in late 1953 with Dexter Tisby and Bruce Tate. Williams, who was a former member of the Hollywood Flames, brought along a song called Earth Angel which the group recorded with Duncan on lead vocals.
Dootone Records picked up the record, along with the song Hey Senorita, and released a single in late 1954. While Earth Angel was the b-side, a DJ flipped the record over to feature the now classic Doo Wop song. The song shot to number 1 on the R&B and number 8 on the Pop chart, ending up the group's biggest hit.
Buck Ram picked up the group, hot off their hit, and brokered a contract with Mercury Records that included the group and the still fledgling Platters. The latter group ended up the hitmakers in the deal as the Penguins never had another national hit except for Pledge of Love which made it to 15 on the R&B chart after they moved to Atlantic Records.
Through the rest of the 50's, the group went through numerous lineup changes with Duncan being the one constant. The remnents of the original group broke up in 1962 but Cleve continued recording under the name with a variety of other performers through the decades. The performed on the PBS Doo Wop 50 special and were inducted into the Vocal Group Hall of Fame in 2004.