让多模态智能体通过经验与技能持续学习,无需更新参数。
XSkill: Continual Learning from Experience and Skills in Multimodal Agents
- 双流框架分别提取行动级经验与任务级技能
- 在5个基准上优于基线模型,显著提升推理能力
- 适合需要长期进化的智能体系统研究
多模态智能体虽能处理复杂推理任务并使用多种工具,但在开放场景下仍存在工具使用效率低和调度僵化的问题。核心挑战在于如何在不更新参数的前提下,从过往轨迹中持续学习。本文识别出两种互补的可复用知识:经验(提供动作级指导)和技能(提供任务级规划支持)。为此提出XSkill,一种基于视觉观测的双流持续学习框架。在积累阶段,通过视觉引导的摘要与跨路径批判,提炼并整合多路径回放中的经验和技能;在推理阶段,根据当前视觉上下文检索并适配知识,并将使用历史反馈至积累环节,形成持续学习闭环。在涵盖多个领域的五个基准上,使用四种主干模型评估,XSkill始终显著优于仅依赖工具或学习型基线。进一步分析表明,两种知识流在影响智能体推理行为上具有互补作用,且具备优异的零样本泛化能力。
原文摘要 · Abstract (English)
Multimodal agents can now tackle complex reasoning tasks with diverse tools, yet they still suffer from inefficient tool use and inflexible orchestration in open-ended settings. A central challenge is enabling such agents to continually improve without parameter updates by learning from past trajectories. We identify two complementary forms of reusable knowledge essential for this goal: experiences, providing concise action-level guidance for tool selection and decision making, and skills, providing structured task-level guidance for planning and tool use. To this end, we propose XSkill, a dual-stream framework for continual learning from experience and skills in multimodal agents. XSkill grounds both knowledge extraction and retrieval in visual observations. During accumulation, XSkill distills and consolidates experiences and skills from multi-path rollouts via visually grounded summarization and cross-rollout critique. During inference, it retrieves and adapts this knowledge to the current visual context and feeds usage history back into accumulation to form a continual learning loop. Evaluated on five benchmarks across diverse domains with four backbone models, XSkill consistently and substantially outperforms both tool-only and learning-based baselines. Further analysis reveals that the two knowledge streams play complementary roles in influencing the reasoning behaviors of agents and show superior zero-shot generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。