用外部行为反馈替代自我反思,提升大模型推理可靠性
Beyond Introspection: Reinforcing Thinking via Externalist Behavioral Feedback
- 通过分析模型行为轨迹,训练外部判别模型进行纠错
- 在多个推理基准上超越主流自省式方法,显著提升准确率
- 无需标注、轻量高效,适合广泛部署于各类大模型
尽管推理时思考使大语言模型(LLMs)能处理复杂问题,但其基于概率的特性导致思维过程在知识边界附近不可靠或不一致。现有方法尝试让模型自我批判以修正错误,但此类自省会继承原始输出的偏见,即‘内省幻觉’。受动物行为学启发,我们提出外部主义三步框架——蒸馏-强化-推理(DRR)。DRR不依赖模型内省,而是通过评估其可观察行为提供纠正反馈。首先蒸馏推理者的动作轨迹,再训练一个轻量级外部判别模型(DM)。推理时,该模型充当批评者,识别并拒绝可疑推理步骤,迫使LLM舍弃错误路径、探索替代方案,从而在不修改基础模型的前提下提升推理质量。多基准实验表明,本框架显著优于主流自省方法。得益于轻量且无需标注的设计,DRR为提升多种LLMs推理可靠性提供了可扩展、可适配的解决方案。
原文摘要 · Abstract (English)
While inference-time thinking allows Large Language Models (LLMs) to address complex problems, the extended thinking process can be unreliable or inconsistent because of the model's probabilistic nature, especially near its knowledge boundaries. Existing approaches attempt to mitigate this by having the model critique its own reasoning to make corrections. However, such self-critique inherits the same biases of the original output, known as the introspection illusion. Moving beyond such introspection and inspired by core methodologies in ethology, we propose an externalist three-step framework Distillation-Reinforcement-Reasoning (DRR). Rather than relying on a model's introspection, DRR evaluates its observable behaviors to provide corrective feedback. DRR first distills the reasoner's behavioral traces, then trains a lightweight, external Discriminative Model (DM). At inference time, this DM acts as a critic, identifying and rejecting suspicious reasoning steps. This external feedback compels the LLM to discard flawed pathways and explore alternatives, thereby enhancing reasoning quality without altering the base model. Experiments on multiple reasoning benchmarks show that our framework significantly outperforms prominent self-critique methods. Benefiting from a lightweight and annotation-free design, DRR offers a scalable and adaptable solution for improving the reliability of reasoning in a wide range of LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。