让电脑操作智能体用图文结合的技能库,提升复杂任务完成率。
VISUALSKILL: Multimodal Skills for Computer-Use Agents

- 构建分层图文技能库,按主题动态加载内容
- 在两个基准上平均得分0.456,比纯文本技能高8.3分
- 适合需要跨软件、长流程操作的研究者和开发者
计算机使用代理(CUAs)在标准评测中接近人类表现,但在长周期任务和未见软件上仍存在困难。现有技能库仅用文本表示技能,忽视了图形用户界面(GUI)的视觉特性。本文提出VISUALSKILL:一种面向特定应用的分层多模态技能结构,以中心索引组织各主题文件,通过load_topic MCP工具按需获取文本与图像。每个技能通过两阶段流程构建,融合人工文档与实时界面探索。在CUA-World和OSExpert-Eval两个基准上,搭载Claude Opus 4.6的Claude Code CLI代理使用VISUALSKILL后平均得分为0.456,较无技能基线(0.303)提升15.3分;相比仅用相同内容生成但无图像的文本技能(0.373),再提升8.3分,证明保留视觉图示有助于代理识别界面元素并验证操作状态。
原文摘要 · Abstract (English)
Computer-use agents (CUAs) approach human-level performance on standardised benchmarks but still struggle on long-horizon tasks and unseen software. Existing skill libraries address this with reusable skills, but represent the skill artifact as text only, despite the visual nature of GUI interaction. We propose VISUALSKILL: a hierarchical multimodal skill, tailored to each target application and organised as a central index over per-topic files, which the agent consumes through a load_topic MCP tool that fetches the relevant topic's text and figures on demand. We construct each skill with a two-stage pipeline that combines authored documentation with live-application UI exploration. On two CUA benchmarks, CUA-World and OSExpert-Eval, a Claude Code CLI agent backed by Claude Opus 4.6 reaches an average score of 0.456 with VISUALSKILL, a +15.3 point absolute lift over the no-skill baseline (0.303). Against a matched text-only skill that is generated from the same source content and differs from VISUALSKILL only in modality, VISUALSKILL yields a further +8.3 point absolute gain over the matched text-only skill (0.373 vs. 0.456), providing direct evidence that retaining visual figures in the skill artifact, rather than verbalizing them away, helps the agent both identify UI elements and verify workflow state after each action. Our code is available at https://github.com/XMHZZ2018/VisualSkills.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。