Could a computer mark a NAPLAN essay?

By Oscar Schwartz

If student assessment is automated, what might it miss?

In October last year, a three-minute story on ABC Radio’s AM program started with a question: “Can a computer mark a piece of writing?” According to Les Perelman, an American academic who taught composition at MIT for two decades, the answer was no. “Automated essay scoring doesn’t work,” he said, because algorithms cannot grasp development, reason and logic in a piece of writing. Perelman’s claim was refuted, with equal confidence, by Stanley Rabinowitz, a PhD in educational psychology and the general manager of the Australian Curriculum, Assessment and Reporting Authority (ACARA). He countered that extensive empirical research had already demonstrated that automated systems can be as “reliable and valid” as human scorers.

Perelman and Rabinowitz were debating algorithmic functionality on national radio because ACARA, the body responsible for administering NAPLAN standardised tests in Australian schools, was planning to roll out automated scoring to assess persuasive or narrative essays for Year 5, 7 and 9 students nationally the following year. ACARA’s plan was to trial the technology by using both computer and human markers in 2018, and comparing their results. If no major discrepancies between algorithm and teacher showed up, they would then move to full automation by 2020. Perelman had been commissioned by the NSW Teachers Federation to review ACARA’s plan and found that it relied on an “extremely dubious methodology and incorrect information”. ACARA’s chair, Steven Schwartz, hit back, claiming that Perelman’s doubts were unfounded. “Computers can analyse x-rays & blood tests, drive cars, take dictation and translate languages,” he tweeted, “but they are unable to mark an 8-year-old’s essay.”

For many teachers, including Dan Yore, this question of algorithmic competence was beside the point. Like many of his colleagues, Yore believes that automated essay scoring would further entrench a pedagogical model that damages the most vital aspect of education: the teacher–student relationship.

Yore teaches a combined class of Years 7 to 9 at Yirrkala Community School in East Arnhem Land. The guiding principle of the school’s pedagogy, according to its director, Merrki Ganambarr-Stubbs, is to provide the 200-odd students with a “both-ways” education – in Yolngu Matha and English, in the classroom and on country. This both-ways approach derives from the concept of garma. When salt water from the ocean and fresh water from the land meet in a river system, a white foam precipitates on the surface. Ganambarr-Stubbs explains that this spume, or garma, is a metaphor for how new forms of knowledge rise out of the meeting of the Yolngu and Australian curricula in the classroom.

Yore first visited the Yirrkala school in 2007 during a medical placement in the community. He was so impressed by the both-ways approach to education that he left the medical profession, enrolled in the Teach for Australia program, moved to the Northern Territory, and worked for a few years as a teacher in Darwin before an opportunity opened up in Yirrkala. After a year of teaching at the school, Yore still believes that the both-ways method is one of the most groundbreaking approaches to pedagogy in Australia. Yet he laments how, for the most part, it is overlooked in national conversations about innovation in education. This is in part due to a common misconception that innovation has to be technological. But it is also because many of the positive outcomes of both-ways education are measured in conversations between teachers and community elders. “When we assess a student’s progress we don’t just look at literacy and numeracy,” he says. “We also speak to elders about whether the student is fulfilling his or her cultural and leadership roles. For me, this is a more accurate and important marker for how a student is tracking than whether they hit certain key criteria in an essay.”

According to Yore, the NAPLAN model of assessment cannot assimilate these qualitative measures of progress. “When it comes to assessing student outcomes on a national scale,” he says, “there is the NAPLAN way or nothing, which means many of the significant benefits of our both-ways education are overlooked or misunderstood.”

Having worked as the data practitioner at his former school, Yore understands the value of standardised data in education. “I know how observations from a distance can help schools pick up patterns and form hypotheses.” His concern is that as national education policy moves further towards an algorithmic model of education, other crucial measures of student development are getting lost. To Yore, the idea of using automated essay scoring is emblematic of this problem. “It is the pinnacle of a reductionist model of education that assesses students entirely outside of their classroom context.”

To see how the both-ways method works in practice, I spent a day in the classroom with Yore and Vanessa, a Yolngu educator who works closely with him. Students arrived between 8 and 8.30am, a total of five girls and two boys. Yore told me it would be a quiet day because there was an important burial ceremony happening in a nearby community, and fulfilling cultural duties is part of the students’ education.

Once everyone had arrived, Yore handed out picture books about the planets and our solar system. The class was in the middle of a term-long unit about the cosmos, which was being taught through the dual lens of Western science and Yolngu mythology. (“Yolngu people have been telling stories about the stars for a long time!” someone had written on the whiteboard.)

After reading time was over, I led a short creativewriting class in which the students had to write a portrait of an undiscovered planet. One girl wrote about a planet made out of bones and hot chocolate rivers. The boys, who were inseparable, imagined a planet called N.W.A where they met an alien called Tupac who had green and red teeth.

The next lesson was maths. Yore handed out worksheets on angles, and the students took out their protractors. There was a big capacity gap in the class, which Yore negotiated by grouping students of similar ability together and then roaming the room to provide one-on-one time with each student.

As the day progressed, lessons became less directive. We watched part of Apollo 13, listened to music in language, and did yoga.

Throughout the day, the way the garma metaphor works in the classroom became more obvious. The relational, both-ways approach influenced everything, from how Yore and Vanessa interacted with each other and the students to the types of posters on the walls. There were English grammar and spelling lists, but also complex matrices outlining the kinship relationships between students. There was a topological world map, but also a huge hand-drawn map that traced a coastal trek the kids had completed earlier in the year with community elders. A collaborative poem written in black marker on a large sheet of butcher’s paper caught my attention. “As busy as a bee / And as ralpa as ants” read one of the lines. (Ralpa is a Yolngu word that is hard to translate, but “industrious” comes closest.)

The idea that software designed to assess linguistic proficiency at a national scale could ever comprehend the dynamic and somewhat chaotic process of Yolngu and English co-creation in this class was absurd.

According to Kate Brady, who works as a teacher in Darwin, using algorithms to mark NAPLAN essays would be just as bad for teachers and students in other schools across the nation. Brady worked as a NAPLAN essay marker in 2014 and 2016. She says that not only did she have the opportunity to learn more about the mechanics of writing and assessment but she also developed a better appreciation of how much effort most students put into their NAPLAN written responses. “I would say that 80 to 90 per cent of the essays I marked showed creativity and personal reflection. If they replaced us with robots, my question would be, where is the appreciation for the effort the child puts in? Where is the empathy and compassion for the student and their ideas and stories?”

On January 25 this year, ACARA announced that it had received instruction from the federal Education Council not to go ahead with automated essay scoring, at least for now. Yore is heartened to hear that people in positions of power are still thinking carefully before introducing new technologies into education. “But to be honest,” he adds, “the fact that we’re even having this debate reminds me of how far off course we’ve come in our pedagogical thinking in this country. It’s not enough to just slightly redirect the ship; we actually have to turn the Titanic around.”