让大模型在强化学习中同时保持准确与探索，通过双模式熵正则化实现。
Policy Split: Incentivizing Dual-Mode Exploration in LLM Reinforcement with Dual-Mode Entropy Regularization
- 将策略分为常规模式与高熵模式，共享参数但目标不同。
- 在各类任务中均超越传统熵引导方法，提升探索多样性。
- 适合需要创造性输出的场景，如内容生成与复杂推理。
为在不损害准确性的前提下促进大语言模型强化学习中的多样化探索，我们提出Policy Split，一种将策略分叉为正常模式与高熵模式的新范式，通过高熵提示实现。两个模式共享模型参数，但分别进行针对性的双模式熵正则化：正常模式优化任务正确性，高熵模式偏好探索，二者协同学习。大量实验表明，该方法在多种模型规模下，于通用与创意任务中持续优于现有熵引导强化学习基线。进一步分析显示，Policy Split实现了双模式探索，高熵模式生成的行为模式与正常模式显著不同，提供独特学习信号。
原文摘要 · Abstract (English)
To encourage diverse exploration in reinforcement learning (RL) for large language models (LLMs) without compromising accuracy, we propose Policy Split, a novel paradigm that bifurcates the policy into normal and high-entropy modes with a high-entropy prompt. While sharing model parameters, the two modes undergo collaborative dual-mode entropy regularization tailored to distinct objectives. Specifically, the normal mode optimizes for task correctness, while the high-entropy mode incorporates a preference for exploration, and the two modes learn collaboratively. Extensive experiments demonstrate that our approach consistently outperforms established entropy-guided RL baselines across various model sizes in general and creative tasks. Further analysis reveals that Policy Split facilitates dual-mode exploration, where the high-entropy mode generates distinct behavioral patterns to the normal mode, providing unique learning signals.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。