Auditing Artificial Intelligence: Context Engineering with Retrieval-Augmented Generation

Abstract

Retrieval-augmented generation (RAG) systems enhance large language models (LLMs) by integrating dynamic external data, enabling more contextually relevant and accurate outputs. As these systems gain traction in accounting applications (e.g., financial reporting, auditing, and tax), concerns emerge regarding data reliability, control oversight, and auditability. This study applies Design Science Research Methodology to develop a practical control framework tailored to the four core stages of a typical RAG system: External Data Management, User Interaction and Query Flow, Retrieval and Prompt Construction, and LLM Inference and Response. The proposed framework includes control objectives linked to stage-specific risks relevant to financial reporting. Iterative validation was conducted using an innovative multi-LLM consensus process to ensure coverage and reduce hallucination risk. This study contributes a framework for evaluating artificial intelligence (AI)-integrated systems, guidance for auditors and regulators to assess probabilistic AI outputs, extension of existing audit control frameworks, and instructional materials for accounting educators.