If you are reading this blog, you might be among the many language teachers who are already using AI tools like ChatGPT in their teaching. But how is AI changing the landscape of language assessment? If you are involved in teaching or assessing English, you have probably heard of automarking. What is it? What are the pros and cons? How should it be used? Will it leave a role for the human examiner with specialist expertise?
I’ve recently co-authored a paper on automarking with colleagues at Cambridge University Press & Assessment. You can read the full paper here, but in this blog I give a brief overview of our findings.
What is automarking?
Thanks to rapid advancements in Machine Learning and Artificial Intelligence, computer algorithms can now be trained to mark written essays and extended speech in exams. This is known as automarking. The advanced technology can predict the scores that human examiners would give, and flag answers that need to be reviewed by a human.
The decisions we make based on assessments that are enabled by AI can significantly impact life opportunities such as education admission, employment or immigration decisions. Automarking is a transformational application of AI in language education. Our research presents the key principles underlying good practice in the use of automarking in L2 language assessment and offers an illustration of how these principles are applied in practice.
What are the benefits?
Automarking offers multiple benefits for both teacher and student:
- Increased speed of marking – much faster than when only human examiners are involved.
- On-demand testing is possible – free from the constraints of examiner availability.
- Consistent performance – well-trained automarking systems can perform to a set high standard over time, while human examiners must be regularly trained to provide reliable ratings.
- Adaptive testing can be enabled –the difficulty of tasks is adjusted throughout the test to test taker performance.
- Integration of learning and assessment – instant automated scores can feed into individual learning paths, based on test results which identify areas for development.
What are the limitations and concerns?
Human examiners fearing redundancy need not panic as language experts have identified the following concerns:
- Authenticity – automarking systems perform best with limited tasks such as a short written response to a question, which are not always representative of real-life language use.
- Authenticity in speaking – communicative language ability also includes tone, turn-taking management, and dealing with communication breakdowns and non-verbal behaviour. These aspects are currently extremely difficult to be marked automatically in a reliable manner.
- Validity – scoring algorithms can handle a relatively narrow range of language features, compared to human marking. Current automarking technology is unable to fully measure features such as complexity, accuracy and fluency, nor implied meaning, argument development and the appropriateness of language use in a social context.
- Accountability – it is difficult to give learners and stakeholders visibility of the assessment criteria used by automarking systems, compared to assessments evaluated by human markers.
- Ethics – the potential for exam malpractice and challenges to exam integrity by strategies to cheat on automarked tests.
- User competence – inadequate AI literacy may lead to misuse of automated language assessment by test takers.
Considerations for using automarking
In our paper we propose 12 fundamental principles that underpin the development, use, and broader contexts of automarked language tests.
Some of these considerations include:
- What data has the automarking system been trained on, and can it ensure no positive or negative bias? The training data must be carefully selected to reflect the targeted test taker population and their various patterns of responses.
- To ensure fair opportunities for all test takers, automarkers need to be continuously evaluated and adjusted in accordance with the changing test taker population.
- What is the accuracy of the automarking system and how is it evaluated? Computer marks should be compared to ‘gold standard’ marks provided by trained and certified human examiners.
- How automarking systems work and what their scores mean must be easily understood by everyone, from regulatory bodies to other stakeholders such as examiners, teachers, learners, and parents. This information needs to be publicly available.
- What is the potential for malpractice? Detection methods need to target and flag a wide range of behaviours which are considered unusual and deviate significantly from expected or typical responses.
- What is the degree of examiner involvement? A hybrid human/machine approach may be best to mitigate the limitations of the technology and uphold high standards of marking.
How Cambridge uses automarking systems
Cambridge uses different automarking systems for different exams, as they are trained on different datasets with different test taker demographics and different task types. The systems are consistent and behave exactly the same way, each time they mark a script.
The systems uses natural language processing (NLP) techniques to extract language features from essays and apply a complex set of scoring rules to these features to evaluate the quality of writing against a set of scoring criteria. The deployed model is trained offline and so does not carry out any further learning or evolution while being used in operational tests. AI systems that learn and evolve on their own, as seen in Generative AI models, might be less suited to automarking test contexts, since there is less transparency in the process and the features underpinning the automarking decisions.
The Cambridge automarking process involves five stages:
- All responses are marked by the trained automarking system, which includes a scoring model, a confidence model and a model to detect aberrant responses.
- Those with a high confidence metric and not flagged as anomalous are released to test takers.
- Marks which do not meet the confidence threshold are sent to our pool of examiners for allocation and marking. Any scripts flagged as potentially anomalous are also sent to human markers for review. In cases of discrepancies between human and automarking system scores, the test taker receives the human score.
- Additionally, a random selection of responses is sent for human marking regardless of the score confidence metric. This helps to ensure examiners are exposed to a full range of scripts, rather than just those flagged by the confidence system, to maintain their standard of marking.
- Human marking is then subject to further statistical monitoring and reviewed as with our current marking practices.
If you are interested in learning more about how Cambridge trains and evaluates its automarking system, find out more here.
What next for automarking technology?
While automarking technology undoubtedly has an increasing role to play in language assessment, it’s important to understand the limitations, constraints and vulnerabilities. Its implementation should be based on research, and our study highlights the importance of blending AI with the expertise of human examiners – we believe it is essential to keep a ‘human in the loop’.
For more information, read the full paper: Automarking in language assessment: Key considerations for best practice
Want more advice on using AI in the English classroom? We’ve got a list of blogs you’ll find interesting!