无需标注数据，通过降低模型预测熵提升大模型推理能力。
Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization
- 在无监督环境下，通过最小化潜在语义空间中的预测熵来激励推理。
- 数学任务上准确率从30.7%提升至48.1%，MMLU-Pro上从32.1%升至50.1%。
- 适合追求零标注训练、可解释性推理优化的研究者与应用开发者。
现有增强大语言模型推理能力的方法主要依赖监督微调（SFT）结合强化学习，在特定推理数据上进行训练。这些方法严重依赖外部监督信号，如标注的推理路径、验证过的正确答案或预训练奖励模型。本文提出熵最小化策略优化（ ours），首次尝试完全无监督的大模型推理激励。通过在潜在语义空间中持续最小化大模型对未标注问题的预测熵， ours 在数学和自由形式自然推理任务上达到与有监督方法相当的效果。具体而言，无需任何监督信号， ours 将 Qwen2.5-Math-7B Base 在数学基准上的准确率从30.7%提升至48.1%，将 Qwen2.5-7B Base 在 MMLU-Pro 上的准确率从32.1%提升至50.1%。实验与分析进一步揭示了 ours 的有效性。代码已开源：https://github.com/QingyangZhang/EMPO。
原文摘要 · Abstract (English)
Existing methods to enhance the reasoning capability of large language models predominantly rely on supervised fine-tuning (SFT) followed by reinforcement learning (RL) on reasoning-specific data. These approaches critically depend on external supervisions--such as labeled reasoning traces, verified golden answers, or pre-trained reward models. In this work, we propose Entropy Minimized Policy Optimization (\ours), which makes an early attempt at fully unsupervised LLM reasoning incentivization. By continuously minimizing the predictive entropy of LLMs on unlabeled questions in a latent semantic space, \ours achieves competitive performance compared to supervised counterparts on both mathematical and free-form natural reasoning tasks. Specifically, without any supervised signals, \ours boosts the accuracy of Qwen2.5-Math-7B Base from 30.7\% to 48.1\% on mathematical benchmarks and improves the accuracy of Qwen2.5-7B Base from 32.1\% to 50.1\% on MMLU-Pro. Primary experiments and analysis are also provided to interpret the effectiveness of \ours. Code is available at https://github.com/QingyangZhang/EMPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。