针对复杂电脑操作数据稀缺问题，构建多模态新基准并合成数据提升模型表现。
Covering Human Action Space for Computer Use: Data Synthesis and Benchmark
- 基于渲染器自动生成五种模态的交互数据，涵盖点击、拖拽、绘图等动作。
- 新基准覆盖长尾复杂操作，现有开源模型在该任务上失败率更高。
- 合成数据训练的模型在复杂任务上优于参数量更小的同类模型，适合研究人机交互。
计算机使用代理（CUAs）可自动化屏幕操作，如GPT-5.4和Claude所示，但在复杂、低频交互上的可靠性仍较差，影响用户信任。我们对先进模型的失败案例分析发现，图形界面操作存在长尾分布特征：少数复杂多样的交互导致了大部分任务失败。我们推测这主要源于复杂交互数据稀缺。为此，提出新基准CUActSpot，评估模型在五种模态（GUI、文本、表格、画布、自然图像）及多种动作（点击、拖拽、绘图等）下的能力，覆盖范围超越以往以点击为中心的基准。设计基于渲染器的数据合成流水线：为每种模态自动生成场景，记录截图与元素坐标，并由大语言模型生成匹配指令与操作轨迹。在该语料库上训练的Phi-Ground-Any-4B模型，在少于320亿参数的开源模型中表现领先。相关数据、代码与模型将公开于https://github.com/microsoft/Phi-Ground.git。
原文摘要 · Abstract (English)
Computer-use agents (CUAs) automate on-screen work, as illustrated by GPT-5.4 and Claude. Yet their reliability on complex, low-frequency interactions is still poor, limiting user trust. Our analysis of failure cases from advanced models suggests a long-tail pattern in GUI operations, where a relatively small fraction of complex and diverse interactions accounts for a disproportionate share of task failures. We hypothesize that this issue largely stems from the scarcity of data for complex interactions. To address this problem, we propose a new benchmark CUActSpot for evaluating models' capabilities on complex interactions across five modalities: GUI, text, table, canvas, and natural image, as well as a variety of actions (click, drag, draw, etc.), covering a broader range of interaction types than prior click-centric benchmarks that focus mainly on GUI widgets. We also design a renderer-based data-synthesis pipeline: scenes are automatically generated for each modality, screenshots and element coordinates are recorded, and an LLM produces matching instructions and action traces. After training on this corpus, our Phi-Ground-Any-4B outperforms open-source models with fewer than 32B parameters. We will release our benchmark, data, code, and models at https://github.com/microsoft/Phi-Ground.git
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。