Barrie Kosky is going to be pissed off. No booing, no heckling, no rotten fruit, no veal cutlets (yes, an Italian audience once threw these after a futurist performance): just whoops and wild applause greeting the performers at the end of the opening night of The Lost Echo, at the Sydney Theatre. Sure, family, friends and colleagues of the cast - there are more than 30 people on stage in the final act - were there en masse, and none was just going to clap politely. Maybe some were applauding out of sheer relief that their eight-hour ordeal was over, but the bitter truth for Kosky, a director who says he loves it when an audience boos, is that the vast majority were on their feet acknowledging a singular moment in Australian theatre. Sorry, Baz.

This is, after all, a hugely ambitious work based on Ovid's Metamorphoses, an epic 15-volume poem which tells the story, through popular mythology (a mix of Greek and Roman), of the creation and early history of the world. The production, like the poem, leaps from one story to another without following any particular narrative strand or set of characters, and deals with the appropriately theatrical theme of transformation.

Kosky and Tom Wright, the writer and dramaturge of The Lost Echo, selected 12 of the stories. But how to rework these massive, imagistic texts for the stage? For that, you need Kosky at the top of his game. For a few years, he has been working the opera circuit in Europe, his greatest success being the 2003 production of Györgi Ligeti's Le Grand Macabre at Berlin's Komische Opera, credited with restoring the fortunes of that venerable institution. Ligeti's "satire", with its grotesque treatment of sex and death ("plenty of S&M, nymphomania and infantilism to spice up the action", as one reviewer put it), invites comparison with The Lost Echo.

The anchorman is Tiresias (played by John Gaden) who, as the blind seer known for having changed sex - when a woman she even managed to have children so (s)he is no ordinary trannie - embodies the idea of transformation. He sets the tone in Act I, narrating the story of the death of Phaeton, son of the sun god, before launching into Noël Coward's ‘Mad About the Boy'. The production swerves wildly between these moments of virtuoso narration, which build intensity, and arrangements of popular songs. This structure of tension and release brings a degree of order to what is otherwise an anarchic combination of theatrical elements.

Kosky oversees the chaos from his little prompter's box at the foot of the stage, engineering the piece's mood swings and providing a piano accompaniment. The pace and tone of his playing - he resembles a crazed Play School piano-man - establishes the rhythm for the performers, some of whom even acknowledge his presence: Gaden gives him a velvet glove and suggests that he not put it on, as it might affect his playing; Marta Dusseldorp (as Linfea) asks him to rehearse her in a musical number that turns into an obscene panto in which she performs messy fellatio on Deborah Mailman's Satirino.

These moments are not incidental: they indicate the broader aesthetic of The Lost Echo. There is a studied casualness to this work, a stagy effortlessness which suggests both the performers' mastery of the material and the logic of the piece. It is a very Australian show not because of the content, nor despite it: it's the way all the ambition and complexity of the work is relentlessly undermined. Kosky is not attempting to enhance his reputation through an authoritative association with one of the core texts of Western culture; he's focused on translating it for a popular audience. This is an active vision of culture - as something that is vital and enacted, rather than inherited and worshipped - and it's the real achievement of The Lost Echo.

There are moments in Act I when you feel you're watching a musical comedy, with the focus on the love triangle of Paul Capsis (as Diana), Peter Carroll (Jove, the deity in chief) and the object of his affections, Callisto, played with gamine cheerfulness by Amber McMahon. Pamela Rabe's entrance as Juno (the wife of Jove), sliding on the sperm-wet stage before pausing to sip her absinthe and deliver a spine-tingling version of Cole Porter's ‘C'est Magnifique', demonstrates a truly Ovidian metamorphosis.

Such lightness of touch gives way to a weightier Act II, which features a sequence of soliloquies delivered down-stage of a mobile glass cubicle. The images Kosky constructs in this cubicle are some of the most unnerving and challenging seen in mainstream Australian theatre in recent years. After Hayley McElhinney delivers Myrrha's story of incest down-stage, venetian blinds open in the glass box to reveal formations of clowns masturbating gargantuan phalluses and ejaculating blood onto the glass walls, to the tune of Cole Porter's ‘Night and Day'.

Following Rabe - spellbinding again as Salmacis, narrating her hermaphrodite's story - a figure in the box vomits milk in time to ‘I've Got You Under My Skin'. The act closes with an extended narration of the story of Philomela, climaxing with an astonishing musical sequence organised around Purcell's ‘Remember Me'. It works beautifully, but what the hell is it about?

Kosky worked with second-year drama students at NIDA for the chorus in Acts I and II, continuing his connection with the country's top acting academy. Early in the process, he suggested that the students read Artaud, which suggests a way of understanding this work. Artaud's ‘Theatre of Cruelty' called for theatre to bring about a radical re-invention of culture. At the centre of it was a disturbing call for a newly configured aesthetics of the body, in which violent and surreal imagery could shock an audience into a heightened, poetic state.

There are clear links between Artaud's vision and Kosky's carnivalesque constructions. Like Artaud, Kosky's theatre is physical and demanding for both performers and spectators. And, in returning to myth as the source for his theatre, Kosky appears to take inspiration from Artaud's belief in the power of art to "awaken the gods that sleep in the museums".

Act III of The Lost Echo stages the brutal and devastating consequences of crossing the gods, in the story of the Bacchae, adapted from Euripides. The admonitions of Tiresias not to put ego and intellect before the honour of the deities fall on deaf ears, and the stubborn young king, Theseus, is torn to pieces by his mother and aunts. Kosky does not spare the audience, relishing the bloodlust of the piece (enacted, with some difficulty, by the younger performers in a filthy public toilet). Act IV then focuses on the story of Orpheus and the song cycle Die Winterreise, by Schubert. The mythic resonances of the work are subordinated to the music, as Orpheus himself, in Peter Carroll's demented characterisation, meanders hopelessly around the stage.

But what of Echo herself? She doesn't feature in the play directly, but her absence is important. She loves Narcissus and prays for death when he spurns her. The gods grant her wish, but also allow her voice to live on after her death. According to Jungian psychology, the echo is a metaphor for the possibility of instinctual reflection and change: it is a self-regulatory mechanism by which nature (including humanity) can understand itself in its own terms, without the prescriptiveness of morality or religion.

Seen in this light, The Lost Echo returns us to an older order of understanding - it repudiates the careful management of feeling and of imagery associated with religion, especially Christianity, in favour of a more instinctual approach. The sheer iconoclastic force of its images reminds us that theatre can provide a radical alternative vision to the increasingly straitjacketed orthodoxies of Australian society. And this isn't just about Kosky's explorations of Ovid's archetypal visions: while he imbues the production with avant-garde vigour, The Lost Echo is also the result of significant institutional investment. Robyn Nevin and the STC Actors Company have backed a great experiment, and I suspect we will hear its resonances for some time.