Using Generative AI to Identify Arguments in Judges’ Reasons: Accuracy and Benefits for Students
-
Monash University
Australia -
Monash University Australia
-
Monash University Australia
-
Monash University Australia
Abstract
This study evaluates the effectiveness of generative artificial intelligence (GAI) in identifying and reconstructing legal arguments from judges’ reasons in court cases, focusing on the practical implications for law students and legal educators. By examining the performance of two versions of popular Large Language Models – ChatGPT and Claude – across five recent High Court of Australia decisions, the study makes a preliminary assessment of the accuracy of LLM systems in replicating a skill essential for lawyers: identification of arguments and argument chains in judges’ reasons. The methodology involves marking LLM-generated outputs with reference to both a sample answer and a detailed rubric.
Key findings reveal a significant variance in the accuracy of different LLMs, with Claude 3.5 markedly outperforming all others, achieving average grades up to 90 per cent. In contrast, ChatGPT versions demonstrated lower accuracy, with average marks not exceeding 50 per cent. These results highlight the critical importance of selecting the right GAI system for legal applications, as well as the necessity for users to critically engage with AI outputs rather than relying solely on automated tools.
The study concludes that while LLMs hold potential benefits for the legal profession, including increased efficiency and enhanced access to justice, for GAI use that may be carried out by a law student, the technology cannot yet replace the nuanced human skill of legal argument analysis.
Downloads
Author Biographies
Paul is a senior lecturer at Monash University. He is interested in the ways in which the law and legal institutions will need to adapt in order to cater for, and understand, AI in all of its guises -- something that he explored in a recent book: AI and the Rule of Law (Hart, 2024). His interdisciplinary research includes projects that: use a language agent to answer legal questions; use LLMs to aid academics in assessing students and designing marking rubrics; and, use generative AI to determine contractual damages.
Iwan Williams is a philosopher at the Monash Centre for Consciousness and Contemplative Studies (M3CS), Monash University. He completed his PhD in late 2021, at the Monash Philosophy department. His research focuses on the nature and structure of representations in minds and machines.
Dr. Lizhen Qu is a lecturer (assistant professor) at the Faculty of Information Technology at Monash University, a founding member of the AIM lab, and an expert in robust and privacy-preserving neuro-symbolic methods for natural language processing (NLP) and multimodal applications. His work seamlessly integrates deep learning, logical reasoning, and causality. In addition, he has conducted extensive research on a wide range of NLP applications, with a special focus on legal AI, digital health, and social NLP. Prior to his current role, he served as a research scientist at Data61/CSIRO, and completed his PhD at Saarland University and the Max-Planck-Institute for Informatics.
AustraliaDr. Teresa Weiqing Wang is currently a senior lecturer (roughly equivalent to the level of "associate professor" in North American universities) in data science with Faculty of Information Technology, Monash University. She previously worked as a post-doctoral research fellow at Data and Knowledge Engineering (DKE) Group, the University of Queensland, where she also obtained her Ph.D. degree. She obtained both her bachelor degree in Software Engineering and master degree in Computer Science from Nanjing University.
