arXiv:2602.08520cs.AIcs.LG2026-02

用模型自身不确定性指导重推理,零成本提升推理准确率。

Reinforcement Inference: Leveraging Uncertainty for Self-Correcting Language Model Reasoning

  • 根据模型生成时的熵值判断是否需要第二次思考
  • 在MMLU-Pro上准确率从60.7%提升至84.0%,仅多1.61次调用
  • 适合追求高精度且容忍轻微延迟的部署场景

现代大语言模型通常采用单次、贪婪解码进行评估与部署,尤其在要求确定行为的专业场景中。这种策略会系统性低估模型真实能力:许多错误源于内部模糊下的过早决断,而非知识缺失。本文提出强化推理(Reinforcement Inference),一种基于熵的推理时控制策略,利用模型自身的不确定性来选择性触发第二次更谨慎的推理过程,实现更强性能而无需重新训练。在14个学科共12,032道MMLU-Pro题目上,使用DeepSeek-v3.2在零样本条件下,确定性解码下准确率从60.72%提升至84.03%,额外推理调用仅增加61.06%。100%重试消融达84.35%,表明不确定性引导的选择已捕获绝大部分可提升收益,且显著降低计算开销。提示词仅优化的消融实验表现低于基线,说明增益并非来自通用提示。本研究不仅提供实用的推理时优化方案,更揭示了以熵为控制信号测量和扩展模型能力的新范式:因解码模型自回归生成,熵与置信度等指标天然成为生成过程中的第一类控制信号。

原文摘要 · Abstract (English)

Modern large language models (LLMs) are often evaluated and deployed under a one-shot, greedy inference protocol, especially in professional settings that require deterministic behavior. This regime can systematically under-estimate a fixed model's true capability: many errors arise not from missing knowledge, but from premature commitment under internal ambiguity. We introduce Reinforcement Inference, an entropy-aware inference-time control strategy that uses the model's own uncertainty to selectively invoke a second, more deliberate reasoning attempt, enabling stronger performance without any retraining. On 12,032 MMLU-Pro questions across 14 subjects, using DeepSeek-v3.2 with deterministic decoding in a zero-shot setting, Reinforcement Inference improves accuracy from 60.72% to 84.03%, while only incurring 61.06% additional inference calls. A 100% re-asking ablation reaches 84.35%, indicating that uncertainty-aware selection captures most of the attainable improvement with substantially less compute. Moreover, a prompt-only ablation underperforms the baseline, suggesting that the gains are not explained by generic prompting alone. Beyond providing a practical inference-time upgrade, our results suggest a broader entropy-aware paradigm for measuring and expanding model capability: because modern decoder-based models generate outputs autoregressively, entropy and related confidence measures arise naturally as first-class control signals during generation. The resulting gap between one-pass greedy inference and uncertainty-conditioned deliberation offers a diagnostic lens on an LLM's latent reasoning horizon and motivates future training objectives that explicitly constrain correctness--confidence alignment.

推理优化不确定性大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。