Caligula formed in Sydney in 1990 and were composed of five members: vocalist Ashley Rothschild, guitarist James McKinnon, drummer Dave Macken, Jamie Fonti (keyboards and backing singer), and bass guitarist Sean Fonti. The band also included Dave Ross and Mitchell Foley as founding members. In 1990 and 1991, the band released two EPs, before a compilation album including both EPs, titled Got One.

In 1994, they released their debut studio album Rubenesque and received national airplay on Triple J and Triple M with the song "Tears of a Clown" (Smokey Robinson & the Miracles cover) which reached #24 in the Australian singles chart.