April 2018

The Nation Reviewed

Could a computer mark a NAPLAN essay?

By Oscar Schwartz

If student assessment is automated, what might it miss?

In October last year, a three-minute story on ABC Radio’s AM program started with a question: “Can a computer mark a piece of writing?” According to Les Perelman, an American academic who taught composition at MIT for two decades, the answer was no. “Automated essay scoring doesn’t work,” he said, because algorithms cannot grasp development, reason and logic in a piece of writing. Perelman’s claim was refuted, with equal confidence, by Stanley Rabinowitz, a PhD in educational psychology and the general manager of the Australian Curriculum, Assessment and Reporting Authority (ACARA). He countered that extensive empirical research had already demonstrated that automated systems can be as “reliable and valid” as human scorers.

Perelman and Rabinowitz were debating algorithmic functionality on national radio because ACARA, the body responsible for administering NAPLAN standardised tests in Australian schools, was planning to roll out automated scoring to assess persuasive or narrative essays for Year 5, 7 and 9 students nationally the following year. ACARA’s plan was to trial the technology by using both computer and human markers in 2018, and comparing their results. If no major discrepancies between algorithm and teacher showed up, they would then move to full automation by 2020. Perelman had been commissioned by the NSW Teachers Federation to review ACARA’s plan and found that it relied on an “extremely dubious methodology and incorrect information”. ACARA’s chair, Steven Schwartz, hit back, claiming that Perelman’s doubts were unfounded. “Computers can analyse x-rays & blood tests, drive cars, take dictation and translate languages,” he tweeted, “but they are unable to mark an 8-year-old’s essay.”

For many teachers, including Dan Yore, this question of algorithmic competence was beside the point. Like many of his colleagues, Yore believes that automated essay scoring would further entrench a pedagogical model that damages the most vital aspect of education: the teacher–student relationship.

Yore teaches a combined class of Years 7 to 9 at Yirrkala Community School in East Arnhem Land. The guiding principle of the school’s pedagogy, according to its director, Merrki Ganambarr-Stubbs, is to provide the 200-odd students with a “both-ways” education – in Yolngu Matha and English, in the classroom and on country. This both-ways approach derives from the concept of garma. When salt water from the ocean and fresh water from the land meet in a river system, a white foam precipitates on the surface. Ganambarr-Stubbs explains that this spume, or garma, is a metaphor for how new forms of knowledge rise out of the meeting of the Yolngu and Australian curricula in the classroom.

Yore first visited the Yirrkala school in 2007 during a medical placement in the community. He was so impressed by the both-ways approach to education that he left the medical profession, enrolled in the Teach for Australia program, moved to the Northern Territory, and worked for a few years as a teacher in Darwin before an opportunity opened up in Yirrkala. After a year of teaching at the school, Yore still believes that the both-ways method is one of the most groundbreaking approaches to pedagogy in Australia. Yet he laments how, for the most part, it is overlooked in national conversations about innovation in education. This is in part due to a common misconception that innovation has to be technological. But it is also because many of the positive outcomes of both-ways education are measured in conversations between teachers and community elders. “When we assess a student’s progress we don’t just look at literacy and numeracy,” he says. “We also speak to elders about whether the student is fulfilling his or her cultural and leadership roles. For me, this is a more accurate and important marker for how a student is tracking than whether they hit certain key criteria in an essay.”

According to Yore, the NAPLAN model of assessment cannot assimilate these qualitative measures of progress. “When it comes to assessing student outcomes on a national scale,” he says, “there is the NAPLAN way or nothing, which means many of the significant benefits of our both-ways education are overlooked or misunderstood.”

Having worked as the data practitioner at his former school, Yore understands the value of standardised data in education. “I know how observations from a distance can help schools pick up patterns and form hypotheses.” His concern is that as national education policy moves further towards an algorithmic model of education, other crucial measures of student development are getting lost. To Yore, the idea of using automated essay scoring is emblematic of this problem. “It is the pinnacle of a reductionist model of education that assesses students entirely outside of their classroom context.”

To see how the both-ways method works in practice, I spent a day in the classroom with Yore and Vanessa, a Yolngu educator who works closely with him. Students arrived between 8 and 8.30am, a total of five girls and two boys. Yore told me it would be a quiet day because there was an important burial ceremony happening in a nearby community, and fulfilling cultural duties is part of the students’ education.

Once everyone had arrived, Yore handed out picture books about the planets and our solar system. The class was in the middle of a term-long unit about the cosmos, which was being taught through the dual lens of Western science and Yolngu mythology. (“Yolngu people have been telling stories about the stars for a long time!” someone had written on the whiteboard.)

After reading time was over, I led a short creativewriting class in which the students had to write a portrait of an undiscovered planet. One girl wrote about a planet made out of bones and hot chocolate rivers. The boys, who were inseparable, imagined a planet called N.W.A where they met an alien called Tupac who had green and red teeth.

The next lesson was maths. Yore handed out worksheets on angles, and the students took out their protractors. There was a big capacity gap in the class, which Yore negotiated by grouping students of similar ability together and then roaming the room to provide one-on-one time with each student.

As the day progressed, lessons became less directive. We watched part of Apollo 13, listened to music in language, and did yoga.

Throughout the day, the way the garma metaphor works in the classroom became more obvious. The relational, both-ways approach influenced everything, from how Yore and Vanessa interacted with each other and the students to the types of posters on the walls. There were English grammar and spelling lists, but also complex matrices outlining the kinship relationships between students. There was a topological world map, but also a huge hand-drawn map that traced a coastal trek the kids had completed earlier in the year with community elders. A collaborative poem written in black marker on a large sheet of butcher’s paper caught my attention. “As busy as a bee / And as ralpa as ants” read one of the lines. (Ralpa is a Yolngu word that is hard to translate, but “industrious” comes closest.)

The idea that software designed to assess linguistic proficiency at a national scale could ever comprehend the dynamic and somewhat chaotic process of Yolngu and English co-creation in this class was absurd.

According to Kate Brady, who works as a teacher in Darwin, using algorithms to mark NAPLAN essays would be just as bad for teachers and students in other schools across the nation. Brady worked as a NAPLAN essay marker in 2014 and 2016. She says that not only did she have the opportunity to learn more about the mechanics of writing and assessment but she also developed a better appreciation of how much effort most students put into their NAPLAN written responses. “I would say that 80 to 90 per cent of the essays I marked showed creativity and personal reflection. If they replaced us with robots, my question would be, where is the appreciation for the effort the child puts in? Where is the empathy and compassion for the student and their ideas and stories?”

On January 25 this year, ACARA announced that it had received instruction from the federal Education Council not to go ahead with automated essay scoring, at least for now. Yore is heartened to hear that people in positions of power are still thinking carefully before introducing new technologies into education. “But to be honest,” he adds, “the fact that we’re even having this debate reminds me of how far off course we’ve come in our pedagogical thinking in this country. It’s not enough to just slightly redirect the ship; we actually have to turn the Titanic around.”

Oscar Schwartz

Oscar Schwartz is a Melbourne-based writer.

View Edition

From the front page

Image of ‘Bewilderment’

‘Bewilderment’ by Richard Powers

The Pulitzer winner’s open-hearted reworking of Flowers for Algernon, updated for modern times

Image of Prime Minister Scott Morrison and Deputy Prime Minister Barnaby Joyce. Images via ABC News

Morrison’s mandate

Barnaby Joyce acknowledges that a net-zero target is cabinet’s call. But what exactly is their mandate?

Image of ‘Scary Monsters’

‘Scary Monsters’ by Michelle de Kretser

Two satirical stories about fitting in, from the two-time Miles Franklin–winner

Image of Jeremy Strong as Kendall Roy in HBO’s Succession season 3. Photograph by David Russell/HBO

Ties that bind: ‘Succession’ season three

Jeremy Strong’s performance in the HBO drama’s third season is masterful

In This Issue

Tutu Bob of Kings Cross

A local tour guide proves there is still plenty of life in the Cross

Image of David Wotherspoon

Sick on the inside

Our corrective services struggle to cope with the mental health requirements of inmates

Image of Marshall Islands, 1946

Nuclear brinkmanship and the doomsday scenario

The risk posed by the global weapons complex is much worse than you know

Image of Jeremy Heimans

Jeremy Heimans: the up-start

The co-founder of GetUp! might be the most influential Australian in the world

More in The Nation Reviewed

Illustration by Jeff Fisher

Close to home for Katy Gallagher

Life in quarantine as COVID-19 hits Senator Katy Gallagher’s family

Illustration by Jeff Fisher

A loss of character

Remembering some of Sydney’s well-known streetfolk

Illustration by Jeff Fisher

The agony and ecstasy

Clinical trials in Perth will study the use of MDMA to treat PTSD and addiction

Illustration by Jeff Fisher

Return of the devil

Tasmanian devils may soon be returning to the wild on the mainland

Read on

Image of Jeremy Strong as Kendall Roy in HBO’s Succession season 3. Photograph by David Russell/HBO

Ties that bind: ‘Succession’ season three

Jeremy Strong’s performance in the HBO drama’s third season is masterful

Image of a tampon and a sanitary pad viewed from above

A bloody shame: Paid period leave should be law

Australia’s workplace laws must better accommodate the reproductive body

Image of Gladys Berejiklian appearing before an ICAC hearing in October 2020. Image via ABC News

The cult of Gladys Berejiklian

What explains the hero-worship of the former NSW premier?

Cover image of ‘Bodies of Light’

‘Bodies of Light’ by Jennifer Down

The Australian author’s latest novel, dissecting trauma, fails to realise its epic ambitions