轻量级GUI代理通过自适应感知实现快慢推理切换，兼顾效率与精度。
iSHIFT: Lightweight Slow-Fast GUI Agent with Adaptive Perception
- 引入隐式思维链与感知控制模块，动态切换快慢推理模式。
- 2.5B参数模型在多个基准上达到顶尖性能，兼具高效与高精度。
- 适合需要精确界面交互的轻量化自动化任务，如UI测试与智能助手。
多模态大语言模型（MLLMs）在解读和交互复杂的像素密集型图形用户界面（GUI）环境方面展现出巨大潜力。然而，构建既高效完成高层任务又精准执行细粒度交互的智能体仍具挑战性。GUI智能体需高效执行常规操作，同时处理依赖精确视觉定位的任务，但现有方法在识别特定界面元素时表现不佳。此外，这些MLLMs仍规模庞大，无法根据任务需求自适应调整推理深度。本文提出iSHIFT：一种基于灵活令牌的隐式慢-快混合推理轻量级智能体，结合隐式思维链与感知控制模块。iSHIFT使MLLM能在慢速模式下利用详细视觉定位实现高精度，或在快速模式下使用全局线索提升效率。特殊感知令牌引导注意力至相关屏幕区域，使模型自主决定推理方式与关注位置。尽管仅2.5B参数，iSHIFT在多个基准数据集上达到当前最优性能。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) show strong potential for interpreting and interacting with complex, pixel-rich Graphical User Interface (GUI) environments. However, building agents that are both efficient for high-level tasks and precise for fine-grained interactions remains challenging. GUI agents must perform routine actions efficiently while also handling tasks that demand exact visual grounding, yet existing approaches struggle when accuracy depends on identifying specific interface elements. These MLLMs also remain large and cannot adapt their reasoning depth to the task at hand. In this work, we introduce iSHIFT: Implicit Slow-fast Hybrid Inference with Flexible Tokens, a lightweight agent that integrates latent thinking (implicit chain-of-thought) with a perception control module. iSHIFT enables an MLLM to switch between a slow mode, which leverages detailed visual grounding for high precision and a fast mode that uses global cues for efficiency. Special perception tokens guide attention to relevant screen regions, allowing the model to decide both how to reason and where to focus. Despite its compact 2.5B size, iSHIFT matches state-of-the-art performance on multiple benchmark datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。