arbiter
JournalsAccount

A Machine Learning and Large Language Model-Integrated Approach to Research Project Evaluation

Jian Ma et al.

Journal of Database Management2024https://doi.org/10.4018/jdm.345400article
AJG 1ABDC A
Weight
0.55

Abstract

Research project evaluation upon completion is one of the important tasks for research management in government funding agencies and research institutions. Due to the increased number of funded projects, it is hard to find qualified reviewers in the same research disciplines. This paper proposes a machine learning and large language model integrated approach to provide decision support for research project evaluation. Machine learning algorithms are proposed to compute the weights of key performance indicators (KPIs) and scores of KPIs based on the evaluation results of completed projects, large language models are used to summarize research contributions or findings on project reports. Then domain experts are invited to consolidate the weights and scores for the KPIs and assess the novelty and impact of research contribution or findings. Experiments have been conducted in practical settings and the results have shown that the proposed method can greatly improve research management efficiency and provide more consistent evaluation results on funded research projects.

4 citations

Open via your library →

Cite this paper

https://doi.org/https://doi.org/10.4018/jdm.345400

Or copy a formatted citation

@article{jian2024,
  title        = {{A Machine Learning and Large Language Model-Integrated Approach to Research Project Evaluation}},
  author       = {Jian Ma et al.},
  journal      = {Journal of Database Management},
  year         = {2024},
  doi          = {https://doi.org/https://doi.org/10.4018/jdm.345400},
}

Paste directly into BibTeX, Zotero, or your reference manager.

Flag this paper

A Machine Learning and Large Language Model-Integrated Approach to Research Project Evaluation

Flags are reviewed by the Arbiter methodology team within 5 business days.

Evidence weight

0.55

Balanced mode · F 0.40 / M 0.15 / V 0.05 / R 0.40

F · citation impact0.59 × 0.4 = 0.24
M · momentum0.60 × 0.15 = 0.09
V · venue signal0.50 × 0.05 = 0.03
R · text relevance †0.50 × 0.4 = 0.20

† Text relevance is estimated at 0.50 on the detail page — for your query’s actual relevance score, open this paper from a search result.