让大模型智能体在复杂任务中更精准地分配奖励,提升决策质量。
GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillation

- 通过自蒸馏对比学生与教师,动态识别任务中的关键转折点。
- 在偏离教师轨迹时自动划分适应性段落,精细调整局部奖励权重。
- 在数学推理和工具使用任务中显著优于传统方法,尤其适合困难长序列任务。
强化学习已成为大模型智能体常用的后训练方法,但通常依赖结果级奖励,监督粒度粗糙。尽管细粒度信用分配有望提升策略优化效果,但如何可靠获取局部信用并精准分配到长序列中的特定部分仍是难题。本文提出粒度自适应优势重加权框架GEAR,利用自蒸馏生成的词元级与片段级信号,重构轨迹级GRPO优势。GEAR通过比较在线策略的学生与基于真值条件的教师,获得参考引导的偏差信号,用于识别自适应片段边界,并调节局部优势权重。该偏差常在语义偏离起始处激增,而后续自回归延续的词元可能重新趋于低偏差。GEAR将此类激增视为锚点,当学生与教师保持一致时保留词元级分辨率;一旦偏离,则将对应延续划分为自适应片段,并以偏离点的偏差值调节该片段的优势。在八个数学推理与代理工具使用基准上,使用Qwen3 4B与8B模型的实验表明,GEAR持续优于标准GRPO、仅自蒸馏基线以及词元或回合级信用分配方法。在基线性能较低的任务中提升尤为显著,最高可达比GRPO高出约20%,说明该自适应重加权机制在更具挑战性的长序列场景中尤为有效。
原文摘要 · Abstract (English)
Reinforcement learning has become a widely used post-training approach for LLM agents, where training commonly relies on outcome-level rewards that provide only coarse supervision. While finer-grained credit assignment is promising for effective policy updates, obtaining reliable local credit and assigning it to the right parts of the long-horizon trajectory remains an open challenge. In this paper, we propose Granularity-adaptivE Advantage Reweighting (GEAR), an adaptive-granularity credit assignment framework that reshapes the trajectory-level GRPO advantage using token- and segment-level signals derived from self-distillation. GEAR compares an on-policy student with a ground-truth-conditioned teacher to obtain a reference-guided divergence signal for identifying adaptive segment boundaries and modulating local advantage weights. This divergence often spikes at the onset of a semantic deviation, while later tokens in the same autoregressive continuation may return to low divergence. GEAR therefore treats such spikes as anchors for adaptive credit regions: where the student remains aligned with the teacher, token-level resolution is preserved; where it departs, GEAR groups the corresponding continuation into an adaptive segment and uses the divergence at the departure point to modulate the segment' s advantage. Experiments across eight mathematical reasoning and agentic tool-use benchmarks with Qwen3 4B and 8B models show that GEAR consistently outperforms standard GRPO, self-distillation-only baselines, and token- or turn-level credit-assignment methods. The gains are especially strong on benchmarks with lower GRPO baseline accuracy, reaching up to around 20\% over GRPO, suggesting that the proposed adaptive reweighting scheme is especially useful in more challenging long-horizon settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。