让温度参数自动学习，提升大模型强化学习的探索效率。
Temperature as a Meta-Policy: Adaptive Temperature in LLM Reinforcement Learning
- 将温度设为可学习的元策略，动态调节生成多样性。
- 在5个数学推理任务上超越固定或启发式温度设定。
- 无需额外采样，直接通过奖励信号在线优化温度分布。
温度是大型语言模型中控制生成过程中探索与利用权衡的关键超参数。高温度促进多样但嘈杂的输出，低温度则导致聚焦但可能过早收敛。然而，静态或启发式温度调度无法适应强化学习训练过程中的动态需求，常限制策略改进。本文提出温度自适应元策略优化（TAMPO），将温度控制重构为可学习的元策略。TAMPO采用分层双循环机制：内环中，基于元策略选定的温度采样轨迹，并使用GRPO等方法更新语言模型策略；外环中，元策略通过奖励高优势轨迹的出现概率，更新候选温度的分布。该轨迹引导、奖励驱动的机制实现无额外采样的在线适应，直接对齐探索与策略优化。在五个数学推理基准测试中，TAMPO显著优于采用固定或启发式温度的基线方法，证明温度可作为大模型强化学习中有效且可学习的自适应探索元策略。论文已获ICLR 2026接收。
原文摘要 · Abstract (English)
Temperature is a crucial hyperparameter in large language models (LLMs), controlling the trade-off between exploration and exploitation during text generation. High temperatures encourage diverse but noisy outputs, while low temperatures produce focused outputs but may cause premature convergence. Yet static or heuristic temperature schedules fail to adapt to the dynamic demands of reinforcement learning (RL) throughout training, often limiting policy improvement. We propose Temperature Adaptive Meta Policy Optimization (TAMPO), a new framework that recasts temperature control as a learnable meta-policy. TAMPO operates through a hierarchical two-loop process. In the inner loop, the LLM policy is updated (e.g., using GRPO) with trajectories sampled at the temperature selected by the meta-policy. In the outer loop, meta-policy updates the distribution over candidate temperatures by rewarding those that maximize the likelihood of high-advantage trajectories. This trajectory-guided, reward-driven mechanism enables online adaptation without additional rollouts, directly aligning exploration with policy improvement. On five mathematical reasoning benchmarks, TAMPO outperforms baselines using fixed or heuristic temperatures, establishing temperature as an effective learnable meta-policy for adaptive exploration in LLM reinforcement learning. Accepted at ICLR 2026.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。