arXiv:2608.26991cs.AI2026-08中稿 · EMNLP

用结构化指令替代截图点击,让AI更高效操作软件。

ASIL: Replacing Screenshot-and-Click with Structured State and Semantic Actions

论文配图:ASIL: Replacing Screenshot-and-Click with Structured State and Semantic Actions
图 1 · 摘自论文原文
  • 用JSON状态和可执行语义动作构建软件接口
  • 单任务平均少于5步即可完成,成功率超80%
  • 适合需要长期规划的自动化任务,尤其适合训练小模型

强大的代码代理能执行脚本、调用工具和管理文件,但许多重要应用仍主要通过图形用户界面访问。我们指出,截图点击对软件操作代理而言效率低下:截图信息不完整,GUI操作脆弱、语义弱且难以支持长程规划。为此,我们提出ASIL(Agent-Software Interaction Layer),一种面向代理的接口,通过结构化JSON观测和可执行的语义动作暴露软件功能,实现每个应用可达的最深层访问路径。我们在15个应用上实例化ASIL,构建包含300个单应用和80个多应用任务的基准。使用闭源模型时,ASIL在多数任务中达成超过80%的成功率,每任务执行动作少于5次。在修复运行环境和50步截图预算下,相同任务的严格成功率分别为6.6%和26.6%,在更简单的OSWorld类任务中提升至15.0%和53.3%。与应用原生接口对比,在匹配任务上,ASIL比LibreOffice的UNO API高28-38点严格分,但仅达到draw.io的MCP内容契约水平。结构化输入也利于训练:小规模监督微调使Qwen3.5-2B从58.0提升至72.1,Qwen3.5-9B从66.6升至80.4;资源受限的在线策略强化学习进一步将二者提升至74.4和82.2。

原文摘要 · Abstract (English)

Powerful code agents can execute scripts, call tools, and manage files, yet many important applications remain accessible primarily through graphical user interfaces. We argue that screenshot-and-click is an inefficient interface for software-operating agents: screenshots are state-incomplete, and GUI actions are brittle, semantically weak, and poorly matched to long-horizon planning. We introduce ASIL (Agent-Software Interaction Layer), an agent-native interface that exposes software through structured JSON observations and code-executable semantic actions, realized through the deepest feasible access path for each application. We instantiate ASIL across 15 applications and a benchmark of 300 single-application and 80 multi-application tasks. ASIL reaches above 80 with closed models while executing fewer than five actions per task. Under a repaired runtime and a 50-step screenshot budget, the same tasks yield 6.6 and 26.6 strict success under screenshot-and-click control, rising to 15.0 and 53.3 on an easier OSWorld-comparable band. Against application-native interfaces on matched tasks, ASIL exceeds LibreOffice's UNO API by 28-38 strict points but only matches draw.io's MCP content contract. The structured modality also suits training: small-scale SFT raises Qwen3.5-2B from 58.0 to 72.1 and Qwen3.5-9B from 66.6 to 80.4, and resource-limited on-policy RL further raises them to 74.4 and 82.2.

AI代理界面交互自动化结构化输入

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。