Persuasive Legal Writing Using Large Language Models
Damian Curran et al.
What the paper says
Persuasive writing is an essential skill for a lawyer. Budding lawyers hone their persuasive writing skills during their studies, in part through essay-style examinations. However, Large Language Models (LLMs) have recently proved adept at a broad range of language tasks. They could undermine the utility of many existing forms of law school assessment by allowing students to generate essays artificially. This paper explores the extent to which those concerns are warranted. This study first explores the constituent elements of persuasive legal writing and reviews the available literature on an LLM’s competence in each area. It then evaluates whether OpenAI’s powerful LLM, GPT-4, can produce essay-style answers to a post-graduate law school exam on legal theory. The GPT-4 output is compared to essays written by actual honors students by having them blind-graded by human graders using the subject’s examination rubric. The study finds that GPT-4 cannot match the honors students. Whilst it can produce essays which are of a passable grade, there are significant challenges producing higher quality content. The paper closes with observations about the experience, prompt engineering, LLM-bias and the technology’s implications for the legal profession.
1 citation
Evidence weight
Balanced mode · F 0.40 / M 0.15 / V 0.05 / R 0.40
| F · citation impact | 0.49 × 0.4 = 0.20 |
| M · momentum | 0.53 × 0.15 = 0.08 |
| V · venue signal | 0.50 × 0.05 = 0.03 |
| R · text relevance † | 0.50 × 0.4 = 0.20 |
† Text relevance is estimated at 0.50 on the detail page — for your query’s actual relevance score, open this paper from a search result.