让大模型学会像人一样思考，提升自动评分能力。
Think-J: Learning to Think for Generative LLM-as-a-Judge
- 通过少量标注数据训练初始思维能力，再用强化学习优化判断逻辑。
- 在多个数据集上超越生成式与分类器式评分模型，无需额外人工标注。
- 适合需要高质量自动评估的LLM研究者与开发者使用。
LLM-as-a-Judge 指的是对大型语言模型生成的回答进行偏好建模，对 LLM 评估和奖励建模具有重要意义。尽管生成式 LLM 在各类任务中取得显著进展，但其作为评判者的表现仍不尽如人意。本文提出 Think-J，通过学习如何思考来提升生成式 LLM 作为评判者的能力。我们首先利用少量精选数据使模型具备初步判断思维能力；随后基于强化学习（RL）优化判断过程。提出了两种优化方法：离线方法需训练一个判别模型以构建正负样本用于学习；在线方法则采用规则化奖励作为反馈进行优化。实验表明，该方法能显著提升生成式 LLM-Judge 的评估能力，在多个基准上超越生成式与分类器式模型，且无需额外人工标注。
原文摘要 · Abstract (English)
LLM-as-a-Judge refers to the automatic modeling of preferences for responses generated by Large Language Models (LLMs), which is of significant importance for both LLM evaluation and reward modeling. Although generative LLMs have made substantial progress in various tasks, their performance as LLM-Judge still falls short of expectations. In this work, we propose Think-J, which improves generative LLM-as-a-Judge by learning how to think. We first utilized a small amount of curated data to develop the model with initial judgment thinking capabilities. Subsequently, we optimize the judgment thinking traces based on reinforcement learning (RL). We propose two methods for judgment thinking optimization, based on offline and online RL, respectively. The offline method requires training a critic model to construct positive and negative examples for learning. The online method defines rule-based reward as feedback for optimization. Experimental results showed that our approach can significantly enhance the evaluation capability of generative LLM-Judge, surpassing both generative and classifier-based LLM-Judge without requiring extra human annotations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。