让视觉智能体学会看图决策,通过多模态技能提升任务执行能力。
MMSkills: Towards Multimodal Skills for General Visual Agents

- 用文本+状态卡+多视角关键帧构建可复用的多模态技能包
- 从公开交互轨迹中自动生成技能,提升视觉代理通用性
- 适合需要复杂视觉推理的GUI和游戏类智能体研究者
可复用技能已成为提升智能体能力的核心基础,但现有技能包多以文本提示、可执行代码或学习到的行为序列为主。对于视觉智能体而言,程序性知识本质上是多模态的:重用不仅依赖于执行什么操作,还涉及识别相关状态、解读视觉进展或失败证据,并决定下一步动作。我们提出多模态程序性知识的形式化定义,并解决三大实际挑战:(I)多模态技能包应包含什么内容;(II)如何从公开交互经验中获取这些技能;(III)推理时如何利用多模态证据而不依赖过多图像上下文或过度锚定参考截图。我们提出MMSkills框架,用于表示、生成和使用运行时视觉决策的可复用多模态过程。每个MMSkill是一个紧凑的状态条件型包,包含文本流程、运行时状态卡和多视图关键帧。为构建这些技能,我们开发了基于智能体的轨迹转技能生成器,通过工作流分组、流程归纳、视觉定位和元技能引导审计,将公共非评估轨迹转化为可复用多模态技能。使用时,引入分支加载的多模态技能代理:选择的状态卡与关键帧在临时分支中检查,与实时环境对齐,并提炼为结构化指导输入主代理。在GUI和游戏类视觉代理基准上的实验表明,MMSkills持续提升前沿及小型多模态智能体性能,表明外部多模态程序性知识可有效补充模型内部先验。
原文摘要 · Abstract (English)
Reusable skills have become a core substrate for improving agent capabilities, yet most existing skill packages encode reusable behavior primarily as textual prompts, executable code, or learned routines. For visual agents, however, procedural knowledge is inherently multimodal: reuse depends not only on what operation to perform, but also on recognizing the relevant state, interpreting visual evidence of progress or failure, and deciding what to do next. We formalize this requirement as multimodal procedural knowledge and address three practical challenges: (I) what a multimodal skill package should contain; (II) where such packages can be derived from public interaction experience; and (III) how agents can consult multimodal evidence at inference time without excessive image context or over-anchoring to reference screenshots. We introduce MMSkills, a framework for representing, generating, and using reusable multimodal procedures for runtime visual decision making. Each MMSkill is a compact, state-conditioned package that couples a textual procedure with runtime state cards and multi-view keyframes. To construct these packages, we develop an agentic trajectory-to-skill Generator that transforms public non-evaluation trajectories into reusable multimodal skills through workflow grouping, procedure induction, visual grounding, and meta-skill-guided auditing. To use them, we introduce a branch-loaded multimodal skill agent: selected state cards and keyframes are inspected in a temporary branch, aligned with the live environment, and distilled into structured guidance for the main agent. Experiments across GUI and game-based visual-agent benchmarks show that MMSkills consistently improve both frontier and smaller multimodal agents, suggesting that external multimodal procedural knowledge complements model-internal priors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。