arXiv:2604.13019cs.CV2026-04

让代码编辑器光标定位更精准,通过多次尝试自动纠错。

PrecisionCUA: Iterative Visual Refinement for Pixel-Precise Cursor Grounding in Code Editors

论文配图:PrecisionCUA: Iterative Visual Refinement for Pixel-Precise Cursor Grounding in Code Editors
图 1 · 摘自论文原文
  • 用多轮视觉反馈迭代修正光标位置,实现像素级精准定位。
  • 在复杂代码界面中,点击准确率和任务成功率显著优于单次预测模型。
  • 适合开发智能编程助手或自动化测试工具的研究者与工程师。

计算机使用代理(CUAs)依赖图形用户界面(GUI)定位将语言指令转化为可执行的屏幕操作,但在密集代码界面(如 VS Code、Cursor)中,对子像素精度的光标定位仍缺乏研究。现有方法多采用单次坐标预测,缺乏错误修正机制,难以应对高密度界面。本文开展了一项关于代码环境中像素级光标定位的实证研究,提出一种基于视觉反馈的多轮迭代精化机制,使代理能根据前序尝试结果自我修正偏差并适应动态界面变化。我们在 Claude、Qwen、GPT 等模型上评估该方法,在一系列复杂编码基准测试中,多轮精化显著优于当前最优的单次预测模型,在点击精度和任务完成率上均取得提升。结果表明,迭代视觉推理是下一代可靠软件工程代理的关键能力。代码已开源:https://github.com/microsoft/precision-cua-bench/tree/main。

原文摘要 · Abstract (English)

Computer Use Agents (CUAs) fundamentally rely on graphical user interface (GUI) grounding to translate language instructions into executable screen actions, but editing-level grounding in dense coding interfaces (such as VS Code and Cursor), where sub-pixel accuracy is required to interact with dense IDE elements, remains underexplored. Existing approaches typically rely on single-shot coordinate prediction, which lacks a mechanism for error correction and often fails in high-density interfaces. In this technical report, we conduct an empirical study of pixel-precise cursor localization in coding environments. Instead of a single-step execution, our agent engages in an iterative refinement process, utilizing visual feedback from previous attempts to reach the target element. This closed-loop grounding mechanism allows the agent to self-correct displacement errors and adapt to dynamic UI changes. We evaluate our approach across Claude, Qwen, and GPT on a suite of complex coding benchmarks, demonstrating that multi-turn refinement significantly outperforms state-of-the-art single-shot models in both click precision and overall task success rate. Our results suggest that iterative visual reasoning is a critical component for the next generation of reliable software engineering agents. Code: https://github.com/microsoft/precision-cua-bench/tree/main.

代码生成视觉定位智能代理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。