通过去除界面截图中的冗余视觉块，提升计算机操作智能体的效率与性能。
ReVision: Scaling Computer-Use Agents via Temporal Visual Redundancy Reduction
- 用学习的补丁选择器对比连续截图，保留空间结构同时去重。
- 在三个基准上平均减少46%视觉标记量，成功率提升3%。
- 适合需要长历史记忆的自动化交互任务研究者使用。
计算机使用智能体（CUAs）依赖图形用户界面的视觉观测，每个屏幕截图会被编码为大量视觉标记。随着交互轨迹增长，标记开销迅速上升，在固定上下文和计算预算下限制了可纳入的历史信息量。这导致与其他领域不同，使用历史信息无法带来明显性能提升。本文提出ReVision，通过训练多模态语言模型处理轨迹时，利用学习的补丁选择器去除连续截图间的冗余视觉块，同时保留模型所需的空间结构。在OSWorld、WebTailBench和AgentNetBench三个基准测试中，使用Qwen2.5-VL-7B处理包含5个历史截图的轨迹时，ReVision平均降低46%的标记用量，且成功率相比无丢弃基线提升3%。这一效率提升使智能体能以更少标记处理更长轨迹。基于此，我们重新评估历史的作用，发现当冗余被消除后，更多历史观测仍能持续提升性能。
原文摘要 · Abstract (English)
Computer-use agents (CUAs) rely on visual observations of graphical user interfaces, where each screenshot is encoded into a large number of visual tokens. As interaction trajectories grow, the token cost increases rapidly, limiting the amount of history that can be incorporated under fixed context and compute budgets. This has resulted in no or very limited improvement in the performance when using history unlike other domains. We address this inefficiency by introducing ReVision, which is used to train multimodal language models on trajectories where redundant visual patches are removed using a learned patch selector that compares patch representations across consecutive screenshots while preserving spatial structure required by the model. Across three benchmarks, OSWorld, WebTailBench, and AgentNetBench, when processing trajectories with 5 history screenshots using Qwen2.5-VL-7B, ReVision reduces token usage by 46% on average while improving success rate by 3% over the no drop baseline. This establishes a clear efficiency gain, enabling agents to process longer trajectories with fewer tokens. With this improved efficiency, we revisit the role of history in CUAs and find that performance continues to improve as more past observations are incorporated when redundancy is removed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。