通过模型自生成轨迹挖掘探索信号，提升语言模型强化学习的推理能力。
ExTra: Exploratory Trajectory Optimization for Language Model Reinforcement Learning
- 从模型自身输出中提取多样性奖励和熵引导的中间步骤重生成。
- 在6个数学推理任务上，pass@1提升5分，pass@16提升7分。
- 适合需要增强推理覆盖范围和准确率的研究者。
基于可验证奖励的语言模型强化学习在任务难度两极均会失效：简单提示常导致全正确、低多样性的推演组，梯度信号不足；复杂提示则可能产生全错误组，无正向奖励。本文提出ExTra（探索性轨迹优化），一种兼容GRPO的框架，从模型自身推演中提取探索信号。ExTra结合两种机制：(i) 在GRPO归一化后加入基于嵌入的多样性奖励，鼓励多样化正确解；(ii) 基于熵信号评分部分轨迹，并从有前景的中间步骤继续探索。在六个数学推理基准测试中，ExTra使Qwen3-1.7B相比GRPO在pass@1上提升约5分，在pass@16上提升约7分，表明轨迹级探索信号能同时提升单样本准确率与推理时覆盖率。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) for language-model reasoning can fail at both extremes of task difficulty: easy prompts often produce all-correct, low-diversity rollout groups with little gradient signal, while hard prompts can produce all-incorrect groups with no positive reward. We introduce ExTra (Exploratory Trajectory Optimization), a GRPO-compatible framework that extracts exploration signals from the model's own rollouts. ExTra combines two mechanisms: (i) a novelty reward that adds embedding-based diversity bonuses after GRPO normalization, rewarding diverse correct solutions; and (ii) entropy-guided prefix regeneration, which scores partial trajectories using entropy signals and continues exploration from promising intermediate steps. Across six mathematical reasoning benchmarks, ExTra improves Qwen3-1.7B over GRPO by about +5 points on pass@1 and +7 points on pass@16, showing that trajectory-level exploration signals can improve both single-sample accuracy and inference-time coverage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。