arXiv:2607.13854cs.CL2026-07中稿 · ACM MM 2026被引 1

让智能体在训练中同时进化技能和策略,提升多模态任务表现。

SPyCE: Skill-Policy Co-evolution for Multimodal Agents

论文配图:SPyCE: Skill-Policy Co-evolution for Multimodal Agents
图 1 · 摘自论文原文
  • 将推理轨迹提炼为分层技能库,与策略共同演化。
  • 在8个基准上超越强化学习与记忆基方法,性能更优。
  • 适合构建能持续学习的复杂多模态智能体。

多模态智能体通过图像迭代操作视觉证据并跨步骤调用工具。现有强化学习方法将轨迹简化为标量奖励,迫使策略在每个新任务中从头发现可复用的工具使用模式;基于记忆的方法虽保留过往经验,但依赖测试时检索,无法更新策略以吸收这些模式。本文核心洞察是:多模态推理轨迹应被提炼为可复用的技能,并在训练中与策略协同进化,而非作为奖励消耗或静态存储检索。为此提出SPyCE(Skill-Policy Co-evolution)框架,将轨迹提炼为分层技能库并在强化学习过程中持续更新。执行技能捕捉局部视觉操作,工作流技能编码高层先验以协调工具使用。训练中,策略基于检索到的技能指导推演,而技能库则利用策略生成的高价值推演进行演化。这形成闭环:更优策略生成更佳技能,不断演化的技能库又为策略提供更强先验。八个基准实验表明,SPyCE始终优于基于RL与记忆的基线。进一步分析显示,分层技能设计与协同演化机制对性能至关重要。结果表明,联合技能-策略优化是构建强大多模态智能体的有前景范式。

原文摘要 · Abstract (English)

Multimodal agents that think with images iteratively manipulate visual evidence and invoke tools across many steps. Existing reinforcement learning methods reduce trajectories to scalar rewards, forcing the policy to discover reusable tool-use patterns from scratch on every new task; memory-based alternatives retain past experience, yet they rely on test-time retrieval, without updating the policy to absorb reusable patterns from that experience. Our key insight is that multimodal reasoning trajectories should be distilled into reusable skills that co-evolve with the policy during training, rather than being consumed as rewards or retrieved from a static store. To this end, we propose SPyCE (Skill-Policy Co-evolution), a framework that distills trajectories into a hierarchical skill library and updates it throughout reinforcement learning. Execution skills capture local visual operations, while workflow skills encode high-level priors that orchestrate tool use. During training, the policy model conditions on retrieved skills to guide its rollouts, while the skill library evolves using valuable rollouts generated by the policy. This creates a closed loop in which improved policies yield better skills, and the evolving skill library, in turn, provides stronger priors for policy rollouts. Experiments across eight benchmarks demonstrate that SPyCE consistently outperforms both RL-based and memory-based baselines. Further analysis reveals that both the hierarchical skill design and the co-evolution mechanism are critical to our design. These results suggest joint skill-policy optimization as a promising paradigm for building capable multimodal agents.

多模态技能进化强化学习智能体

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。