arbiter
JournalsAccount

Understanding Rater Cognition in Performance Assessment: A Mixed IRTree Approach

Hung‐Yu Huang

Applied Psychological Measurement2025https://doi.org/10.1177/01466216251333578article
AJG 2ABDC B
Weight
0.37

Abstract

When rater-mediated assessments are conducted, human raters often appraise the performance of ratees. However, challenges arise regarding the validity of raters' judgments in reflecting ratees' competencies according to scoring rubrics. Research on rater cognition suggests that both impersonal judgments and personal preferences can influence raters' judgmental processes. This study introduces a mixed IRTree-based model for rater judgments (MIM-R), which identifies professional and novice raters by sequentially applying the ideal-point and dominance item response theory (IRT) models to the cognitive process of raters. The simulation results demonstrate a satisfactory recovery of MIM-R parameters and highlight the importance of considering the mixed nature of raters in the rating process, as neglecting this leads to more biased estimations with an increasing proportion of novice raters. An empirical example of a creativity assessment is presented to illustrate the application and implications of MIM-R.

1 citation

Open via your library →

Cite this paper

https://doi.org/https://doi.org/10.1177/01466216251333578

Or copy a formatted citation

@article{hung‐yu2025,
  title        = {{Understanding Rater Cognition in Performance Assessment: A Mixed IRTree Approach}},
  author       = {Hung‐Yu Huang},
  journal      = {Applied Psychological Measurement},
  year         = {2025},
  doi          = {https://doi.org/https://doi.org/10.1177/01466216251333578},
}

Paste directly into BibTeX, Zotero, or your reference manager.

Flag this paper

Understanding Rater Cognition in Performance Assessment: A Mixed IRTree Approach

Flags are reviewed by the Arbiter methodology team within 5 business days.

Evidence weight

0.37

Balanced mode · F 0.40 / M 0.15 / V 0.05 / R 0.40

F · citation impact0.16 × 0.4 = 0.06
M · momentum0.53 × 0.15 = 0.08
V · venue signal0.50 × 0.05 = 0.03
R · text relevance †0.50 × 0.4 = 0.20

† Text relevance is estimated at 0.50 on the detail page — for your query’s actual relevance score, open this paper from a search result.