用对比学习提升大模型论文评价能力,避免单一打分偏差。
From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluation
- 通过对比而非独立打分,让模型学更可靠的学术判断
- 在5个新数据集上平均比基线高21.8%的相对性能
- 适合需要公平、可泛化论文评估的研究者
当前大语言模型对科学论文的评价多采用独立打分方式,但不同会议、时期和标准下的分数尺度差异导致模型易受特定上下文干扰,难以形成稳健的学术判断。为此,我们提出从孤立打分转向协作排序,设计了面向论文评价的比较原生框架CNPE。该框架在数据构建与模型训练中均融入比较机制:首先基于图的相似度排序算法,从论文集合中采样更具信息量的对比对;再通过监督微调与基于对比奖励的强化学习,增强相对质量判断能力。推理阶段,模型对采样论文对进行两两比较,并聚合偏好信号生成全局排名。实验表明,该框架相较强基线DeepReview-14B平均提升21.8%,且在五个未见数据集上表现稳健。代码已开源。
原文摘要 · Abstract (English)
Large language models (LLMs) are currently applied to scientific paper evaluation by assigning an absolute score to each paper independently. However, since score scales vary across conferences, time periods, and evaluation criteria, models trained on absolute scores are prone to fitting narrow, context-specific rules rather than developing robust scholarly judgment. To overcome this limitation, we propose shifting paper evaluation from isolated scoring to collaborative ranking. In particular, we design a $\textbf{C}$omparison-$\textbf{N}$ative framework for $\textbf{P}$aper $\textbf{E}$valuation ($\textbf{CNPE}$), integrating comparison into both data construction and model learning. We first propose a graph-based similarity ranking algorithm to facilitate the sampling of more informative and discriminative paper pairs from a collection. We then enhance relative quality judgment through supervised fine-tuning and reinforcement learning with comparison-based rewards. At inference, the model performs pairwise comparisons over sampled paper pairs and aggregates these preference signals into a global relative quality ranking. Experimental results demonstrate that our framework achieves an average relative improvement of 21.8% over the strong baseline DeepReview-14B, while exhibiting robust generalization to five previously unseen datasets. Our code is available at https://github.com/ECNU-Text-Computing/ComparisonReview.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。