用AI评估英国高校科研质量,能减少评审负担。
Reducing research bureaucracy in UK higher education: Can generative AI assist with the internal evaluation of quality?
- 用ChatGPT模拟评审,对比机构真实结果验证效果。
- 822篇论文中,69%的评分与4*标准一致,边界清晰。
- 适合关注科研评估效率、政策改革的研究者。
本文探讨生成式人工智能(GenAI)在英国高等教育内部科研质量评估中的应用潜力,尤其针对研究卓越框架(REF)的准备过程。基于可行系统模型中的功能替代视角,采用实验方法,利用ChatGPT对REF 2021提交的商管类论文进行评分与排序,并通过“逆向工程”对比AI评分与已知机构结果。在11所机构共822篇论文的严格测试中,发现评分边界与实际REF结果高度一致:1*至2*间占比49%,2*至3*间为59%,3*至4*间达69%。结果表明,AI可提供稳定评估,有效识别临界案例,需人工复核,同时显著降低传统内部评审的巨大资源投入。研究主张采用精细化混合模式,在保障学术严谨性的前提下,应对科研评估官僚成本超百万英镑的问题。尽管存在潜在偏见等局限,该研究为更高效、一致的评估提供了可行框架,有望重塑当前科研评价方式。
原文摘要 · Abstract (English)
This paper examines the potential for generative artificial intelligence (GenAI) to assist with internal review processes for research quality evaluations in UK higher education and particularly in preparation for the Research Excellence Framework (REF). Using the lens of function substitution in the Viable Systems Model, we present an experimental methodology using ChatGPT to score and rank business and management papers from REF 2021 submissions, "reverse engineering" the assessment by comparing AI-generated scores with known institutional results. Through rigourous testing of 822 papers across 11 institutions, we established scoring boundaries that aligned with reported REF outcomes: 49% between 1* and 2*, 59% between 2* and 3*, and 69% between 3* and 4*. The results demonstrate that AI can provide consistent evaluations that help identify borderline evaluation cases requiring additional human scrutiny while reducing the substantial resource burden of traditional internal review processes. We argue for application through a nuanced hybrid approach that maintains academic integrity while addressing the multi-million pound costs associated with research evaluation bureaucracy. While acknowledging these limitations including potential AI biases, the research presents a promising framework for more efficient, consistent evaluations that could transform current approaches to research assessment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。