用一致性推理提升小模型指导大模型的能力，省钱又有效。
CLARity: Reasoning Consistency Alone Can Teach Reinforced Experts
- 用小模型自动检测并奖励逻辑一致的回复，不依赖昂贵标注。
- 在少量数据上提升推理一致性16.5%，准确率提高7.5%。
- 适合资源有限但需高质量推理的领域专家模型训练。
在数据稀缺领域训练专家级大模型困难，常依赖多选题（MCQs）。但标准基于结果的强化学习（RL）虽可能提升准确率，却常损害逻辑一致性。现有监督推理的方法如大规模过程奖励模型（PRMs）成本过高。为此，我们提出CLARity，一种低成本的强化学习框架，仅使用一个小型通用大模型即可提升推理质量。CLARity结合一致性感知奖励机制、两阶段精炼-监控训练流程，以及动态数据重构策略，更高效利用有限数据。实验表明，相较于基线，其响应一致性提升16.5%，准确率提升7.5%。人工评估也证实回复在连贯性与专业性上全面改善。因此，CLARity提供了一种可泛化的方案，使小型模型通过一致性推理有效引导专家模型。代码已开源：https://github.com/Infinite-set/CLARity。
原文摘要 · Abstract (English)
Training expert LLMs in domains with scarce data is difficult, often relying on multiple-choice questions (MCQs). However, standard outcome-based reinforcement learning (RL) on MCQs is risky. While it may improve accuracy, we observe it often degrades reasoning quality such as logical consistency. Existing solutions to supervise reasoning, such as large-scale Process Reward Models (PRMs), are prohibitively expensive. To address this, we propose CLARity, a cost-effective RL framework that enhances reasoning quality using only a small, general-purpose LLM. CLARity integrates a consistency-aware reward mechanism with a 2-stage refine-then-monitor training pipeline to enhance reasoning consistency, and a dynamic data reformulation strategy to to better exploit limited data. Experiments demonstrate that CLARity improves response consistency by 16.5% and accuracy by 7.5% over baselines. Human evaluations further confirm holistic improvements in coherence and professionalism. Thus, CLARity offers a generalizable solution that enables smaller models to effectively guide expert models by reasoning consistency. Our code is open sourced at: https://github.com/Infinite-set/CLARity
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。