视觉语言模型实时代理,自进化降本增效。
VisualClaw: A Real-Time, Personalized Agent for the Physical World

- 分层筛选视频帧+技能库压缩,降低推理开销
- 失败反馈驱动技能迭代,平均准确率提升3.85%至15.80%
- 适配边缘设备,单小时视频仅需5-20次调用
视觉语言模型虽能处理复杂多模态任务,但部署仍存在三大瓶颈:高延迟与成本、代理能力静态化、传统视频问答基准无法检验工具使用场景中的视觉证据利用能力。本文提出VisualClaw,一种基于双原则的自演化多模态代理。首先,混合编码通过级联门控过滤低信息流帧,并采用热/冷top-k注入压缩文本技能库以降低部署成本;其次,技能演化机制让代理从失败中学习:检索记忆作为直接上下文或引导性证据,生成技能库更新以支持未来任务。在4个视频问答基准上,相较于全帧上传,VisualClaw平均减少98%的每问题API成本,相比离线均匀采样8帧基线降低25.9%成本,同时多数场景下准确率提升,如Gemini 3 Flash在EgoSchema上平均提升3.85%,峰值达15.80%。为填补评估空白,我们构建了包含200个场景的VisualClawArena基准,严格遵循五阶段流程;模型需在工作区中结合视频证据、文档、动态更新和可执行检查。在该基准上,相同框架配合计算机使用代理后端,使Codex(GPT-5.5)宏准确率提升2.9%,Claude Code(Sonnet 4.6)提升3.2%,且较均匀采样基线节省9.5%成本。这些特性使VisualClaw特别适合边缘应用,级联机制将一小时流式会话的API调用从约3600次降至仅5-20次,自演化使其成为理想的个性化助手。
原文摘要 · Abstract (English)
Vision language models are serving as general-purpose interfaces for complex multimodal tasks. However, deployment still faces three gaps: VLMs typically incur high latency and cost when processing dense video frames and long prompts, the agent scaffold remains static after deployment, and standard video-QA benchmarks do not test whether agents can use visual evidence inside tool-using workspaces. We present VisualClaw, a self-evolving multimodal agent built around two principles. First, hybrid encoding reduces deployment cost by filtering less informative streaming frames with a cascaded gate and compressing the text skill bank through hot/cold top-k injection. Second, skill evolution lets the agent learn from failures: retrieved memories condition an evolver as direct concatenated context or as guided evidence, producing skill-bank updates that help future questions. Across 4 video-QA benchmarks with 2 VLMs, VisualClaw cuts per-question API cost by an average -98% versus full-frame upload and by -25.9% over the offline uniform 8 frame baseline, while boosting accuracy in most settings, e.g., an average +3.85% and a peak +15.80% on EgoSchema with Gemini 3 Flash. To address the gap, we curate VisualClawArena, a 200-scenario multimodal agentic benchmark built through a strict five-stage pipeline; models must use video evidence, documents, dynamic updates, and executable checks inside a workspace. On VisualClawArena, the same framework with computer-use agent backends improves macro accuracy by +2.9% for Codex (GPT-5.5) and +3.2% for Claude Code (Sonnet 4.6) over no-evolution baselines, with a -9.5% cost reduction compared to the uniform-sampled baseline. These properties make VisualClaw a natural fit for edge applications, where the cascade reduces a 1-hour streaming session from ~3,600 API uploads down to only 5-20 calls and the self-evolution makes it a perfect personalized assistant.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。