By the time the Sunday service is half over, the traffic is just a murmur. During the week you can't hear yourself think at this end of Melbourne's Latrobe Street, much less pray: outside are four lanes of vehicles, two tram lines and a parking garage next door. But on Sundays there are no office workers, no construction workers, hardly any shoppers. Besides, in 1871 church walls were built two-feet thick, as if cold stone alone could protect the faithful from the fires of damnation.

The church could have been larger, grander. In 1854, the Victorian government granted the Welsh Presbyterian Church two acres here - something not uncommon until the State Aid to Religion Abolition Act 1871 stopped such generous gifts from the Crown. There was a condition to the grant: the church had to fence off the allotted land. In the 1850s, though, most able-bodied labouring men were seeking their fortune outside Melbourne, and the pay offered by the church was paltry compared with the riches to be gained from striking gold. One acre was all that was fenced; the other the government reclaimed.

It is a humble building with straight lines and smooth surfaces, befitting the Calvinist tradition. In January 1872, the Argus newspaper said its style was "what is known as the early decorated", an example of the English Gothic revival possessing a "fine axed bluestone base" and a roof "50 feet high to the top of the finial on the gable". Inside, the church is small, with carved, curved pews in a slight incline toward the street. At one end is a Fincham & Sons organ, noted by the National Trust for its regional significance. The other wall is almost all window, tall and intricate, vivid reds and greens, lilies and crosses as sparkling and vibrant as neon. The Argus described it in utilitarian fashion: "large, triple-light window with geometrical tracery filled with stained glass, from the manufactory of Messrs Ferguson, Urie, and Lyon".

By the mid nineteenth century, only around 1800 convicts, or little more than 1% of the total transported to Australia, had come from Wales. Some were rebels and trade unionists involved in uprisings like the one in Merthyr, where workers rioted against their conditions. But free immigration from Wales boomed during the gold rush, when any mining experience, even if it was in coal, seemed like a licence to print money. In 1851, 400 Victorian settlers were Welsh-born; by 1871, the year before the church opened, the number had risen to 7000. The goldfields are still a centre of Welsh culture: there is a sister Welsh Presbyterian church in Sebastopol, near Ballarat, and the older graveyards there have headstones written in Welsh.

Where Welsh churches once dotted Victoria, the Sebastapol and Latrobe Street buildings are now the only ones remaining. Fewer than 5000 Victorians are Welsh-born, and only a tiny percentage of them speak Welsh at home. There are some Welsh-nationalist organisations active in Melbourne, and the Cambrian Society, but it is the city's lauded Welsh choirs and the Latrobe Street church that most publicly keep the culture alive.

The church's congregation today is not only made up of the Welsh and their descendents from around Melbourne and beyond: Asian students who live nearby worship here, as do tourists staying in this part of the city. In the past, the church was the centre of the congregation's social life: there was a tea club, a badminton club, a ladies' basketball team, soccer and social clubs, a dramatic society. "Because people live further and further out, those weekday activities have mostly ceased," church elder and archivist Evan Hughes tells me. Hughes is 92, could pass for 20 years younger, and has worshipped here since 1923, though he doesn't speak Welsh.

The Reverend Siôn Gough Hughes does; he's a passionate and jocular preacher, interspersing his sermons with humorous asides, and with Rugby scores when Wales is victorious and pointed silence when it's not. "I had nine churches in Wales, all with ageing populations," he says. The decision to come here was "a no-brainer". In Melbourne, "the Welsh-language service hasn't changed in 154 years. It's Welsher than a lot of the churches in Wales." The service, attended by 50 to 60 people and the only one of its kind in Australia, is held twice a month. At a time when 90% of the world's languages are each spoken by fewer than 100,000 people, and rare languages are at risk of extinction because of the voracious expansion of English and Mandarin, Spanish and Russian, Welsh is - just - holding its own in the centre of Melbourne.

Much is superb about this small building - its role in other aspects of Australia's history, for one. In the mid 1870s, Emma Constance Stone, a Tasmanian-born governess in her twenties, decided to become a doctor. Women were permitted to study at the University of Melbourne by then, but not in the medical faculty. Eventually Stone graduated from a medical school in Pennsylvania and became the first woman registered as a doctor in Australia. The editor of the Australian Medical Journal would later describe professional women as "beings whom men do not love and with whom [other] women can hardly sympathise" and female doctors as curiosities in the manner of "dancing dogs, fat boys and bearded ladies". But Stone would not be swayed. Her husband, Dr David Egryn Jones, was the minister at the Welsh Presbyterian Church, and in a small hall at the back of the building Stone, together with some of Melbourne University's first female medical graduates, began to treat thousands of poor female patients at a free clinic that later evolved into the Queen Victoria Hospital.

Also superb, elder Bronwen Holding tells me, is the church's music, so central to Welsh culture and religious practice. "Come and see us if you really want to sing hymns," she says. In Melbourne there are four Welsh choirs, and yet the church gives them a run for their money. "A hymn might be three verses long, but somebody decides to sing the last verse again so we all sing the last verse again," Holding says. "It's an automatic response to the feeling. It's the music that hits the soul." It is perhaps also hiraeth, a Welsh word that has no equivalent in English: a longing for your past, or a homesickness, or the sweet pain that is a yearning for the familiar.