让智能体学会识别界面开关状态，准确执行开关指令。
See, Think, Act: Teaching Multimodal Agents to Effectively Interact with GUI by Identifying Toggles
- 通过感知当前状态与指令意图，动态判断是否需要操作
- 在基准测试中使开关指令执行准确率提升超30%
- 适用于需要精准交互的自动化任务场景
多模态智能体在图形界面（GUI）控制中表现出色，但其对开关类指令的执行仍不可靠。为此，我们基于公开数据集构建了一个包含二元开关指令的状态控制基准。现有智能体在当前状态已符合目标时表现尤为不稳定。针对此问题，我们提出状态感知推理（StaR）方法，使智能体能感知当前状态、从指令推断目标状态并作出相应动作。在四个多模态智能体上的实验表明，StaR可将开关指令执行准确率提升超过30%。在三个公开代理基准上的进一步评估显示，StaR也能提升通用任务性能。动态环境测试表明其具备真实应用潜力。代码与基准：https://github.com/ZrW00/StaR。
原文摘要 · Abstract (English)
The advent of multimodal agents facilitates effective interaction within graphical user interface (GUI), especially in ubiquitous GUI control. However, their inability to reliably execute toggle control instructions remains a key bottleneck. To investigate this, we construct a state control benchmark with binary toggle instructions derived from public datasets. Evaluation results of existing agents demonstrate their notable unreliability, particularly when the current toggle state already matches the desired state. To address the challenge, we propose State-aware Reasoning (StaR), a multimodal reasoning method that enables agents to perceive the current toggle state, infer the desired state from the instruction, and act accordingly. Experiments on four multimodal agents demonstrate that StaR can improve toggle instruction execution accuracy by over 30\%. Further evaluations on three public agentic benchmarks show that StaR also enhances general agentic task performance. Finally, evaluations on a dynamic environment highlight the potential of StaR for real-world applications. Code and benchmark: https://github.com/ZrW00/StaR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。