Developing NOVA – A Next-Generation Open Vocabulary Assessment
Ulrich Schroeders & Priscilla Achaa-Amankwaa
Abstract
Abstract: In psychological assessment, vocabulary tests are commonly used as reliable and efficient indicators of crystallized intelligence, as retrospective proxies for premorbid intelligence, and as measures of language proficiency. However, many of the widely used German vocabulary tests are outdated, proprietary, and lack a clear rationale for item selection. To address these limitations, we developed a new, openly available vocabulary test: the Next-Generation Open Vocabulary Assessment (NOVA). We constructed 110 multiple-choice vocabulary items with support from ChatGPT and administered them to 1,052 German-speaking adults using a multiple-matrix design, along with a declarative knowledge test for validation purposes. Using Ant Colony Optimization, we assembled two parallel 30-item short forms by optimizing reliability as well as item difficulty and discrimination parameters within a three-parameter logistic item response model. The resulting test forms provided unidimensional and reliable measurement, covered a broad ability range, showed no gender differences, and correlated strongly with declarative knowledge. A Shiny app is available to calculate norm-referenced scores based on individual test results. Additional analyses revealed that 61.2% of the variance in item difficulty was explained by word frequency and word length, underscoring their potential utility in guiding future vocabulary test development.
Evidence weight
Balanced mode · F 0.40 / M 0.15 / V 0.05 / R 0.40
| F · citation impact | 0.50 × 0.4 = 0.20 |
| M · momentum | 0.50 × 0.15 = 0.07 |
| V · venue signal | 0.50 × 0.05 = 0.03 |
| R · text relevance † | 0.50 × 0.4 = 0.20 |
† Text relevance is estimated at 0.50 on the detail page — for your query’s actual relevance score, open this paper from a search result.