让大模型自己学着调温度，推理更聪明。
Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RL
- 用层级强化学习，根据内部状态动态调整采样温度。
- 数学推理任务上超越固定和启发式温度策略，提升准确率。
- 行为可解释，探索模式与推理不确定性对齐，适合研究生成逻辑的学者。
基于可验证奖励的强化学习（RLVR）通过采样轨迹训练大语言模型，使解码策略成为学习核心而非仅限于推理阶段的选择。采样温度直接影响探索与利用的权衡，通过调节策略熵实现，但现有方法依赖静态值或启发式调整，且与任务级奖励脱节。我们提出内省式大语言模型（Introspective LLM），一个分层强化学习框架，用于在生成过程中学习控制采样温度。在每个解码步骤中，模型根据其隐藏状态选择温度，并从相应分布中采样下一个词元。温度与词元策略通过坐标上升法，联合下游奖励进行优化。在数学推理基准测试中，所学温度策略优于固定值和启发式基线，同时表现出与推理不确定性一致的可解释探索行为。
原文摘要 · Abstract (English)
Reinforcement Learning from Verifiable Rewards (RLVR) trains large language models (LLMs) from sampled trajectories, making decoding strategy a core component of learning rather than a purely inference-time choice. Sampling temperature directly controls the exploration--exploitation trade-off by modulating policy entropy, yet existing methods rely on static values or heuristic adaptations that are decoupled from task-level rewards. We propose Introspective LLM, a hierarchical reinforcement learning framework that learns to control sampling temperature during generation. At each decoding step, the model selects a temperature based on its hidden state and samples the next token from the resulting distribution. Temperature and token policies are jointly optimized from downstream rewards using a coordinate ascent scheme. Experiments on mathematical reasoning benchmarks show that learned temperature policies outperform fixed and heuristic baselines, while exhibiting interpretable exploration behaviors aligned with reasoning uncertainty.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。