用后续问题评估推理链条,让大模型学会更可靠的思考方式。
Self-Training Meets Consistency: Improving LLMs' Reasoning with Consistency-Driven Rationale Evaluation
- 通过追问检测原始推理的可靠性,筛选出更可信的训练样本。
- 在三个数据集上提升推理正确率,逻辑稳健性显著增强。
- 适合想提升大模型思维严谨性的研究人员和开发者。
自训练方法通过让大语言模型学习自身生成的推理过程来提升推理能力。以往方法仅根据最终答案是否正确来判断推理质量,但单一指标易误判,导致模型学习到错误的推理模式。为此,我们提出 CREST(一致性驱动的推理评估自训练框架),通过后续问题进一步评估每个推理过程,并据此指导训练。具体包括:(1) 过滤在后续问题中频繁出错的推理;(2) 基于原始与后续问题的混合偏好进行偏好学习。在三个问答数据集上使用开源大模型的实验表明,CREST 不仅提升了推理的逻辑稳健性和正确性,还优于以往自训练方法。
原文摘要 · Abstract (English)
Self-training approach for large language models (LLMs) improves reasoning abilities by training the models on their self-generated rationales. Previous approaches have labeled rationales that produce correct answers for a given question as appropriate for training. However, a single measure risks misjudging rationale quality, leading the models to learn flawed reasoning patterns. To address this issue, we propose CREST (Consistency-driven Rationale Evaluation for Self-Training), a self-training framework that further evaluates each rationale through follow-up questions and leverages this evaluation to guide its training. Specifically, we introduce two methods: (1) filtering out rationales that frequently result in incorrect answers on follow-up questions and (2) preference learning based on mixed preferences from rationale evaluation results of both original and follow-up questions. Experiments on three question-answering datasets using open LLMs show that CREST not only improves the logical robustness and correctness of rationales but also improves reasoning abilities compared to previous self-training approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。