用数据自动生成细粒度推理错误分类标准,提升大模型对复杂问题的纠错能力。
Generating Data-Driven Reasoning Rubrics for Domain-Adaptive Reward Modeling
- 基于数据构建细粒度推理错误分类体系,指导大模型精准识别错误
- 在代码、数学等专业领域,纠错准确率比基线方法提升45%以上
- 仅需20%的优质标注数据,即可逼近全量标注训练效果,适合高成本领域
大语言模型在验证推理过程时难以可靠识别长文本、专业领域及无明确奖励信号问题中的思维错误。本文提出一种数据驱动的方法,自动构建高度细化的推理错误分类体系(即评分标准),以增强大模型对未见推理轨迹的错误检测能力。实验表明,在编程、数学和化学工程等技术领域,利用这些评分标准的分类方法相比基线方法表现出更强的错误识别能力。该评分体系可用于构建更优的LLM作为裁判的奖励函数,通过强化学习训练推理模型。结果表明,使用这些奖励可使模型在困难任务上的准确率比使用通用大模型作为裁判训练的模型提高45%,且仅需20%的黄金标注数据即可接近由可验证奖励训练模型的性能。本方法将评分标准的应用从评估模型行为质量扩展到评估任务求解的量化正确性,为在缺乏完整黄金标签的情况下训练解决复杂技术问题的模型提供了新路径。
原文摘要 · Abstract (English)
An impediment to using Large Language Models (LLMs) for reasoning output verification is that LLMs struggle to reliably identify errors in thinking traces, particularly in long outputs, domains requiring expert knowledge, and problems without verifiable rewards. We propose a data-driven approach to automatically construct highly granular reasoning error taxonomies to enhance LLM-driven error detection on unseen reasoning traces. Our findings indicate that classification approaches that leverage these error taxonomies, or "rubrics", demonstrate strong error identification compared to baseline methods in technical domains like coding, math, and chemical engineering. These rubrics can be used to build stronger LLM-as-judge reward functions for reasoning model training via reinforcement learning. Experimental results show that these rewards have the potential to improve models' task accuracy on difficult domains over models trained by general LLMs-as-judges by +45%, and approach performance of models trained by verifiable rewards while using as little as 20% as many gold labels. Through our approach, we extend the usage of reward rubrics from assessing qualitative model behavior to assessing quantitative model correctness on tasks typically learned via RLVR rewards. This extension opens the door for teaching models to solve complex technical problems without a full dataset of gold labels, which are often highly costly to procure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。