通过熵机制提升大模型测试时强化学习的探索与利用平衡。
ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanism
- 引入熵驱动的双策略：分叉树多数采样与优势重塑。
- 在AIME 2024上实现68%相对提升，仅用60%采样令牌预算。
- 适合追求高效、高多样性推理的开放域任务研究者。
大语言模型在数学与编程等复杂推理任务中取得显著进展，但仍依赖标注数据，在无监督场景下适应性有限。测试时强化学习（TTRL）通过模型生成伪标签实现自优化，但面临推理成本高和早期估计偏差导致过自信的问题，降低输出多样性并引发性能停滞。为此，我们提出基于熵的机制，包含熵分叉树多数采样（ETMR）与熵基优势重塑（EAR）两种策略。相比基线，该方法使Llama3.1-8B在AIME 2024基准上实现Pass at 1指标68%的相对提升，同时仅消耗60%的采样令牌预算。结果表明，该方法有效平衡了推理效率、输出多样性和估计鲁棒性，推动开放域推理任务中的无监督强化学习发展。
原文摘要 · Abstract (English)
Recent advancements in Large Language Models have yielded significant improvements in complex reasoning tasks such as mathematics and programming. However, these models remain heavily dependent on annotated data and exhibit limited adaptability in unsupervised scenarios. To address these limitations, test-time reinforcement learning (TTRL) has been proposed, which enables self-optimization by leveraging model-generated pseudo-labels. Despite its promise, TTRL faces several key challenges, including high inference costs due to parallel rollouts and early-stage estimation bias that fosters overconfidence, reducing output diversity and causing performance plateaus. To address these challenges, we introduce an entropy-based mechanism to enhance the exploration-exploitation balance in test-time reinforcement learning through two strategies: Entropy-fork Tree Majority Rollout (ETMR) and Entropy-based Advantage Reshaping (EAR). Compared with the baseline, our approach enables Llama3.1-8B to achieve a 68 percent relative improvement in Pass at 1 metric on the AIME 2024 benchmark, while consuming only 60 percent of the rollout tokens budget. This highlights our method's ability to effectively optimize the trade-off between inference efficiency, diversity, and estimation robustness, thereby advancing unsupervised reinforcement learning for open-domain reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。