用指令控制探索，让大模型在强化学习中更主动试错。
Instruction-Conditioned Exploration for Reinforcement Learning with Self-Distillation to an Unconditioned Policy
- 训练时给任务加固定指令，引导模型尝试更多行为
- 使1.7B模型在数学推理上通过率提升5.0%（4K上下文）
- 适合想提升小模型探索能力的研究者
使用强化学习微调大型语言模型已成为提升模型能力的重要手段，但其动作空间结构与传统强化学习不同，导致探索机制面临新挑战。为此，我们提出指令条件探索（ICE），在训练过程中向任务提示添加一组固定的少量指令，统一应用于所有问题，以扩大行为覆盖范围。为支持该方法，我们结合指令条件策略的强化学习与正确轨迹自蒸馏至无条件测试策略。实验表明，该方法使Qwen3-1.7B在4K响应长度下数学推理任务的持有率（pass@1）相比DAPO提升5.0%，且在8K上下文仍有效；而4B模型在4K下未见提升，说明指令扩展了1.7B基模型的行为覆盖。
原文摘要 · Abstract (English)
Post-training Large Language Models (LLMs) with Reinforcement Learning (RL) has become an important tool for improving model capabilities, but the LLM action-space structure introduces challenges distinct from classical RL, with implications for inducing exploration. New methods are required that leverage the broad knowledge and flexibility of pre-trained LLMs to deliberately generate diverse experience at training time. We propose Instruction-Conditioned Exploration (ICE), which appends one of a small fixed set of instructions to task prompts during training, using the same set for every problem, increasing the coverage of behaviours attempted. To facilitate ICE, we combine RL on the instruction-conditioned policy with self-distillation of its correct rollouts into the unconditioned test-time policy. ICE with this objective improves Qwen3-1.7B held-out pass@1 performance at 4K response length on mathematical reasoning tasks by $5.0\%$ relative to training with DAPO, with improvement persisting at a longer 8K context. The improvement does not appear for Qwen3-4B at 4K, where the instructions do not expand base-model coverage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。