仅用熵最小化就能显著提升大模型的推理能力，无需标注数据。
The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning
- 通过降低输出熵，让模型更聚焦于高置信度答案。
- 无监督训练下性能媲美60K标注数据的强化学习基线。
- 推理时调整日志概率，32B模型超越多个闭源大模型。
熵最小化（EM）通过让模型将更多概率集中在最自信的输出上，来提升其表现。我们发现，仅使用这一简单目标，无需任何标注数据，即可显著改善大语言模型在数学、物理和编程等挑战性任务上的表现。研究探索了三种方法：(1) EM-FT，在未标注输出上进行类似指令微调的词元级熵最小化；(2) EM-RL：以负熵为唯一奖励的强化学习；(3) EM-INF：推理时调整日志概率以降低熵，无需训练或参数更新。在 Qwen-7B 上，无标注数据的 EM-RL 表现与使用 60,000 条标注数据的 GRPO、RLOO 等强基线相当甚至更优。此外，EM-INF 使 Qwen-32B 在 SciCode 基准测试中达到或超过 GPT-4o、Claude 3 Opus、Gemini 1.5 Pro 的水平，同时效率比自一致性与顺序精炼高 3 倍。结果表明，许多预训练大模型具备此前被低估的推理能力，仅靠熵最小化即可有效激发，无需标注数据或参数更新。
原文摘要 · Abstract (English)
Entropy minimization (EM) trains the model to concentrate even more probability mass on its most confident outputs. We show that this simple objective alone, without any labeled data, can substantially improve large language models' (LLMs) performance on challenging math, physics, and coding tasks. We explore three approaches: (1) EM-FT minimizes token-level entropy similarly to instruction finetuning, but on unlabeled outputs drawn from the model; (2) EM-RL: reinforcement learning with negative entropy as the only reward to maximize; (3) EM-INF: inference-time logit adjustment to reduce entropy without any training data or parameter updates. On Qwen-7B, EM-RL, without any labeled data, achieves comparable or better performance than strong RL baselines such as GRPO and RLOO that are trained on 60K labeled examples. Furthermore, EM-INF enables Qwen-32B to match or exceed the performance of proprietary models like GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro on the challenging SciCode benchmark, while being 3x more efficient than self-consistency and sequential refinement. Our findings reveal that many pretrained LLMs possess previously underappreciated reasoning capabilities that can be effectively elicited through entropy minimization alone, without any labeled data or even any parameter updates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。