提出新评估框架,让AI更懂专家对论文引言的评价标准
Expert Preference-based Evaluation of Automated Related Work Generation
- 用多轮对话分解引言评价维度,结合对比样例提升判断精度
- 实测显示该框架比普通AI评阅者更贴近人类专家打分
- 适合需要高质量学术写作辅助的科研人员和期刊审稿人
专家级科学写作(如论文引言)高度依赖领域知识。尽管大语言模型在该任务中展现出潜力,但自动评估其生成质量仍是开放难题,因需掌握领域特定标准与专家偏好。传统自动评估指标及通用大模型评分系统难以捕捉此类专业判断。为此,我们以最具挑战性的引言生成为例,提出GREP——一种融合经典评价标准与专家偏好特征的多轮评估框架。该框架将评价拆解为细粒度维度,并引入对比示例提供上下文指导。实证研究显示,相比标准大模型裁判,GREP在评估引言质量时更具鲁棒性,更符合真实科研场景,且与人类专家评分高度相关。同时发现,当前顶尖大模型生成内容仍难以满足合格引言的验证约束。
原文摘要 · Abstract (English)
Expert domain writing, such as scientific writing, typically demands extensive domain knowledge. Although large language models (LLMs) show promising potential in this task, evaluating the quality of automatically generated scientific writing is a crucial open issue, as it requires knowledge of domain-specific criteria and the ability to discern expert preferences. Conventional automatic evaluation metrics and LLM-as-a-judge systems, primarily designed for mainstream NLP tasks, are insufficient to grasp expert preferences and domain-specific quality standards. To address this gap and support realistic human-AI collaborative writing, we focus on related work generation, one of the most challenging scientific tasks, as an exemplar. We propose GREP, a multi-turn evaluation framework that integrates classical related work evaluation criteria with expert-specific preferences. GREP decomposes the evaluation into smaller fine-grained dimensions. This localized evaluation is further augmented with contrastive examples to provide detailed contextual guidance for the evaluation dimensions. Empirical investigation reveals that GREP is able to assess the quality of related work sections in a much more robust manner compared to standard LLM judges, reflects natural scenarios of scientific writing, and bears a strong correlation with the assessment of human experts. We also observe that generations from state-of-the-art LLMs struggle to satisfy validation constraints of a suitable related work section.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。