How close is AI to replacing accounting consultants? Insights from a comparative study of multiple AI models and exit-level accounting students

Abstract

Purpose Artificial intelligence (AI) has the potential to radically transform the accountancy profession, starting with routine transactions and functions and moving towards more strategic, leadership and governance responsibilities. The purpose of this paper is to evaluate the extent to which various Large Language Models (LLMs) can replicate the performance of accounting students completing a professional exit-level examination. By directly comparing the outputs of multiple AI models to those of students pursuing the Chartered Accountant (South Africa) [CA(SA)] designation, the study assesses whether AI has the capacity to support or potentially replace, accounting practitioners. In doing so, it clarifies both the opportunities and limitations of AI for the profession and for accounting education. Design/methodology/approach Exit-level exam questions taken by students pursuing the CA(SA) designation were given to ChatGPT, Claude, CoPilot, Grok and Gemini. A zero-shot prompt method was adopted. Each model was given the scenario and required tasks to complete, with mark allocations, as real-world students were. Basic descriptive statistics and visual representations were used to analyse the data. Findings The results find that AI models are not all equal as far as dealing with financial accounting questions. AI models’ primary strength is in dealing with journal entries and basic calculation questions. They struggle to critically evaluate partial and complete solutions to identify errors and correct them. These tasks require judgement and the ability to distinguish fact from fiction. This is where humans significantly outperform AI and should be where the accountancy profession focuses its student education. Of the AI models, Gemini performed the best overall but did not outperform the student average. Practical implications The study provides a useful baseline for future studies to monitor the progress of AI models in completing technical accounting exams and providing support to accounting professionals. Originality/value The study contributes to the literature by focusing on a renowned global chartered accountant designation from a developing economy and comparing different AI models’ performance to that of exit-level university students. The study also provides objective evidence that speaks to different AI models’ ability to support accounting practitioners in real-world settings or their ability to replace accounting consultants.