让代码编辑器光标定位更精准,通过多次尝试自动纠错。
PrecisionCUA: Iterative Visual Refinement for Pixel-Precise Cursor Grounding in Code Editors

- 用多轮视觉反馈迭代修正光标位置,实现像素级精准定位。
- 在复杂代码界面中,点击准确率和任务成功率显著优于单次预测模型。
- 适合开发智能编程助手或自动化测试工具的研究者与工程师。
计算机使用代理(CUAs)依赖图形用户界面(GUI)定位将语言指令转化为可执行的屏幕操作,但在密集代码界面(如 VS Code、Cursor)中,对子像素精度的光标定位仍缺乏研究。现有方法多采用单次坐标预测,缺乏错误修正机制,难以应对高密度界面。本文开展了一项关于代码环境中像素级光标定位的实证研究,提出一种基于视觉反馈的多轮迭代精化机制,使代理能根据前序尝试结果自我修正偏差并适应动态界面变化。我们在 Claude、Qwen、GPT 等模型上评估该方法,在一系列复杂编码基准测试中,多轮精化显著优于当前最优的单次预测模型,在点击精度和任务完成率上均取得提升。结果表明,迭代视觉推理是下一代可靠软件工程代理的关键能力。代码已开源:https://github.com/microsoft/precision-cua-bench/tree/main。
原文摘要 · Abstract (English)
Computer Use Agents (CUAs) fundamentally rely on graphical user interface (GUI) grounding to translate language instructions into executable screen actions, but editing-level grounding in dense coding interfaces (such as VS Code and Cursor), where sub-pixel accuracy is required to interact with dense IDE elements, remains underexplored. Existing approaches typically rely on single-shot coordinate prediction, which lacks a mechanism for error correction and often fails in high-density interfaces. In this technical report, we conduct an empirical study of pixel-precise cursor localization in coding environments. Instead of a single-step execution, our agent engages in an iterative refinement process, utilizing visual feedback from previous attempts to reach the target element. This closed-loop grounding mechanism allows the agent to self-correct displacement errors and adapt to dynamic UI changes. We evaluate our approach across Claude, Qwen, and GPT on a suite of complex coding benchmarks, demonstrating that multi-turn refinement significantly outperforms state-of-the-art single-shot models in both click precision and overall task success rate. Our results suggest that iterative visual reasoning is a critical component for the next generation of reliable software engineering agents. Code: https://github.com/microsoft/precision-cua-bench/tree/main.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。