解决长程智能体选技能时的信用分配难题,提升决策准确率。
SkillGate: Training In-Policy Skill Selection in Long-Horizon Agents

- 分离选技能与执行的信用通道,避免错误惩罚正确选择。
- 在5个基准上将成功率从40.8%提升至53.2%。
- 适合需要多步决策与技能调用的复杂智能体系统。
智能体框架日益将程序化知识封装为技能:智能体按需读取的指令文件,公共库中已有数千个此类技能。在轨迹中决定读取哪个技能,已成为策略自身需做出的决策,但现有方法缺乏训练信号。我们指出,仅用结果奖励进行强化学习无法教会该决策,根源在于一种结构性问题——选择器信用饥荒:在广播式序列级优势下,仅命名所选技能的少数标记承担几乎可忽略的损失,且随着轨迹延长,其获得的信用符号越来越错。只要后续执行失败,即使选择本身极有价值,也会被惩罚。审计已完成运行的训练痕迹证实了三个性质,均随任务时长单调恶化。SkillGate通过构造性手段消除该问题:将标记支持分为两个独立信用通道,结果信用仅传递给执行标记,而动作局部优势仅传递给技能命名标记,仅当单次读取正确时为正。在16候选集的五个代理基准上,SkillGate使90亿参数模型的试验成功率从40.8%提升至53.2%,显著优于同等预算下的纯结果奖励训练,同时将接触误导候选的概率降低三分之二,读取技能次数更少。
原文摘要 · Abstract (English)
Agent frameworks increasingly package procedural knowledge as skills: instruction files an agent reads on demand, while public libraries now hold thousands of them. Which skill to read has thus become a decision the policy itself makes in the middle of an episode, yet no existing signal trains it. We show that the default remedy, outcome-rewarded RL over the candidate slate, cannot teach it, for a structural reason we identify and name selector credit starvation: under a broadcast, sequence-level advantage, the few tokens that name the chosen skill carry a vanishing share of the loss, and the credit they inherit is increasingly wrong-signed as trajectories lengthen. A correct choice is punished whenever the execution after it fails, even though the choice itself is among the most valuable decisions in the trajectory. Auditing a completed run's own training artifacts confirms all three properties, each worsening monotonically with horizon. SkillGate removes the failure by construction: it partitions the token support into two disjoint credit channels, outcome credit reaching only execution tokens, and a separate action-local advantage reaching exactly the skill-naming tokens, positive only when a trajectory's single read is the correct one. On five agentic benchmarks under a 16-candidate slate, SkillGate lifts a 9B policy from 40.8% to 53.2% trial success, well ahead of the identical budget spent on outcome reward alone, while cutting exposure to misleading candidates by two thirds and reading fewer skills.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。