让电脑使用代理能持续感知屏幕和声音，突破传统截图的局限。
Agent-Computer Observation Interfaces Enable Dynamic Computer Use
- 设计可适配不同模型的感知接口，实现连续观察与离散动作分离。
- 在动态浏览器任务中，准确率提升17至48个百分点，音频任务全解决。
- 适合需要实时感知复杂交互的AI代理开发者或研究者。
SWE-agent首次揭示了软件工程代理中的动作接口是被忽视的设计维度；本文则提出计算机使用（CU）代理中的观察接口同样未被充分探索。当前大多数开源与闭源的CU代理将观察与动作绑定——每3-5秒一张截图，无音频支持——导致其在截图间隙无法感知视频、动画、临时界面事件、会议内容及语音指令。为此，我们提出代理-计算机观察接口（AOI），一个与模型无关的感知层，通过三个门控组件解耦连续自适应观察与离散动作：跨步骤关键帧捕捉、音量触发的音频转录，以及由CU模型生成的视觉叙述文本持久化。静态无声内容下输出极小，不破坏原有循环。在DynaCU-Bench（100个动态浏览器任务+50个静态对照任务）上，7B到前沿规模的CU模型相比截图基线准确率提升17至48个百分点，无需重新训练，原本几乎无法完成的任务变为基本可解。音频任务差距最显著：在语音内容子集上，AOI代理全部完成任务，而流式语音模型虽能准确听清，却无法行动，因缺乏支撑结构。分解分析表明，关键帧选择本身不重要，真正价值在于将捕获帧转化为持久文本叙述；且该接口非固定组合，新模型（Gemini 3 Flash）中关键帧流因图像令牌稀释反而退化，说明组件需按模型定制，不可统一配置。
原文摘要 · Abstract (English)
SWE-agent established the action interface as an underexplored design axis for software-engineering agents; we make the analogous case for the observation interface in computer-use (CU) agents. Current CU agents, closed and open-source alike, tie observation to action--one screenshot every 3-5 s, no audio--leaving them blind and deaf between screenshots to video, animations, transient UI events, meetings, and spoken instructions. We introduce the Agent-Computer Observation Interface (AOI), a model-agnostic perception layer that decouples continuous, adaptive observation from discrete actions through three gated components: inter-step keyframe capture, volume-gated audio transcription, and CU-model-generated visual narration that persists as text. Each produces almost nothing on static, silent content, reducing to the standard loop without degrading it. On DynaCU-Bench (100 dynamic browser tasks plus a 50-task static control), CU models from 7B to frontier scale gain +17 to +48 pp over their screenshot baselines with zero retraining, turning tasks that are near-impossible from periodic screenshots into largely solved ones. The gap is starkest on audio: on a spoken-content subset AOI agents solve every task, whereas streaming voice models hear accurately but cannot act on what they hear without the scaffold. The decomposition is as informative as the headline gain: keyframe selection turns out not to matter--the value comes from narrating captured frames into persistent text--and the interface is not a fixed bundle, since on a newer model (Gemini 3 Flash) the keyframe stream actively regresses through image-token dilution, so its components must be selected per model rather than shipped as one configuration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。