GUI智能体延迟响应?新方法提前预判并即时执行,成功率从50%提至79%
Why Are GUI Agents Correct but Late? Decode on the Decision-Time Critical Path, Tested with Pre-Compiled Policy Trees
- 利用空闲期预构建决策树,事件发生时直接调用无需重算
- 在关键窗口内成功率从50%提升至79%,且零误操作
- 适合可提前枚举动作的场景,特别适合高延迟模型
计算机使用代理常因在瞬态GUI事件中响应过慢而失败,其正确操作往往在窗口关闭后才生成。我们发现根本原因是决策路径上昂贵的自回归解码。为此提出自适应前瞻策略树(AAPT),在不修改底层模型的前提下消除延迟。空闲时段,冻结的多模态模型构建一个有限条件策略树,包含可观测的触发条件、预授权动作及分支专属截止时间,树的规模匹配模型自身解码延迟。事件发生时,轻量级观察器将变化帧匹配到预设分支,并立即执行对应动作,无需生成新文本。成对实验显示,与注册终点对比,在争议决策窗口内成功率达0.79(原为0.50,p=1.8×10⁻³),且无错误动作。开环与预测重规划基线均未成功,因仍于执行时解码。准备时间扫描验证收益出现在延迟树大小预测位置;消融实验证明三个关键:快速观察解码、有效树规划、精准分支路由。注册探针排除初始假设,指向分支路由为因果瓶颈。在126次独立测试中,另一通用多模态模型复现该效果(p=4.9×10⁻¹³)。外部基准测试中,AAPT性能与反应式基线相当,但两者优势互补。结果表明,当候选动作可提前枚举时,AAPT表现最优;否则反应式方法更强。
原文摘要 · Abstract (English)
Computer-use agents often fail on transient GUI events because they produce the correct action only after the relevant window has already closed. We identify the main cause as expensive autoregressive decoding on the decision-time critical path. We propose Adaptive Anticipatory Policy Trees (AAPT), which eliminates this delay without modifying the underlying model. During idle screen periods, the same frozen multimodal model constructs a bounded conditional policy tree with observable guards, pre-authorized actions, and branch-specific deadlines. The tree is sized to cover the model's own decoding latency. When an event occurs, a lightweight observer matches change-gated frames to a prepared branch and immediately executes the corresponding action without generating new text. In paired trials with pre-registered endpoints and exact McNemar tests, AAPT improves the success rate from 0.50 to 0.79 within a contested decision window ($p=1.8\times10^{-3}$), while producing no incorrect actions. Both open-loop and predict-and-replan baselines achieve zero success because they still decode during execution. A preparation-time sweep shows that the gain emerges where the latency-based tree-sizing rule predicts, and ablations reveal three key requirements: fast observer decoding, valid tree planning, and accurate branch routing. A pre-registered oracle probe rejects our initial hypothesis and instead points to branch routing as the causal bottleneck. We further reproduce the effect on an independent general-purpose multimodal model over 126 paired trials ($p=4.9\times10^{-13}$). On an external benchmark, AAPT matches the overall performance of a reactive baseline, although the two methods exhibit complementary strengths. Together, these results suggest that AAPT performs best when candidate actions can be enumerated in advance, whereas reactive execution remains stronger when they cannot.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。