arbiter
JournalsPricingAccount

‘Getting better all the time’: using professional human coach competencies to evaluate the quality of AI coaching agent performance

Jonathan Passmore et al.

Coaching2025https://doi.org/10.1080/17521882.2025.2510643article
AJG 1
Weight
0.52

What the paper says

This paper explores the evaluation of Artificial Intelligence (AI) AI coaching agent (coachbot) as a tool for workplace performance and wellbeing conversations. The paper starts by considering methods for the evaluation of coach performance, including client evaluation, qualitative and quantitative methods and coach competencies. Using a coach competency as a framework, this study uses a qualitative approach to assess a real-world AI coaching agent’s ‘behaviours’ in the form of an assessment by trained ICF assessors of a genuine session with a human client. The study involved a sample of 43 managers who volunteered to be coached by an AI Coaching agent. ICF assessors used the ICF coach competency framework and supporting behavioural anchored rating (BARS) framework developed by the ICF. The results indicated the AI coaching agent was able to demonstrate many elements of both ICF ACC level and PCC level coach competencies. The assessment also revealed gaps in AI coach competence. We conclude AI coaching agents are highly competent at some aspects of the coach process, while less competent at others. Further, AI development is needed to address these gaps.

7 citations

Open paper page →

Cite this paper

https://doi.org/https://doi.org/10.1080/17521882.2025.2510643

Or copy a formatted citation

@article{jonathan2025,
  title        = {{‘Getting better all the time’: using professional human coach competencies to evaluate the quality of AI coaching agent performance}},
  author       = {Jonathan Passmore et al.},
  journal      = {Coaching},
  year         = {2025},
  doi          = {https://doi.org/https://doi.org/10.1080/17521882.2025.2510643},
}

Paste directly into BibTeX, Zotero, or your reference manager.

Flag this paper

‘Getting better all the time’: using professional human coach competencies to evaluate the quality of AI coaching agent performance

Flags are reviewed by the Arbiter methodology team within 5 business days.

Evidence weight

0.52

Balanced mode · F 0.40 / M 0.15 / V 0.05 / R 0.40

F · citation impact0.47 × 0.4 = 0.19
M · momentum0.68 × 0.15 = 0.10
V · venue signal0.50 × 0.05 = 0.03
R · text relevance †0.50 × 0.4 = 0.20

† Text relevance is estimated at 0.50 on the detail page — for your query’s actual relevance score, open this paper from a search result.