小模型无需人工标注即可实现精准界面定位,性能接近大模型。
WinDOM: Self-Family Distillation for Small-Model GUI Grounding
- 用自动化脚本生成5.4万条带框标注数据,无需人工或OCR。
- 自家族蒸馏让20亿参数小模型在跨域任务上提升5.4个百分点。
- 适合移动端部署、无障碍工具开发等轻量级应用需求。
小型(约20亿参数)的GUI定位代理适用于设备端部署、无障碍工具和低成本迭代,但在该规模下仍面临两大开放问题:如何在不依赖昂贵人工标注的情况下获取边界框训练数据,以及如何将监督微调与强化学习结合。本文提出WinDOM,一个通过无头Playwright驱动开源Windows 11网页复现生成的54,425条记录的定位语料库,边界框直接从DOM中读取,无需OCR或人工标注。自家族蒸馏(SFD)是一种仅由教师选择参数化的冷启动方法:可选学生自身指数移动平均(EMA)或冻结的同家族大模型作为教师。我们将SFD冷启动的饱和深度作为显式GRPO超参数。在Qwen3.5-2B学生模型上,未充分饱和的冷启动比收敛后的初始状态更优:使用早期初始化强化学习的SFD-4B相较基线提升5.4个百分点(+3.5 ScreenSpot-Pro,+7.0 OSWorld-G,+5.8 ScreenSpot-V2)。相同规模的EMA模式表现接近跨尺寸40亿参数变体(65.2 vs 66.3),且无需外部教师模型。
原文摘要 · Abstract (English)
Small ($\sim$2B) GUI-grounding agents are attractive for on-device deployment, accessibility tooling, and low-cost iteration, but at this scale they face two open recipe questions: how to obtain bounding-box training data without expensive human annotation, and how to combine supervised fine-tuning with reinforcement learning. We address both, with the explicit goal of pushing small-model performance rather than scaling up. WinDOM is a $54{,}425$-record grounding corpus harvested by driving an open-source Windows 11 web reimplementation under headless Playwright, with bounding boxes read directly off the DOM and no OCR or human annotation. Self-Family Distillation (SFD) is a single rejection-sampling cold-start parameterised only by the teacher choice: either an EMA of the student (no external model) or a frozen larger same-family teacher. We then treat the saturation depth of the SFD cold-start as an explicit GRPO hyperparameter. On a Qwen3.5-2B student, the under-saturated cold-start is a better GRPO initialiser than the converged one: SFD-4B with Early-init RL gains $+5.4$ OOD-mean ($+3.5$ ScreenSpot-Pro, $+7.0$ OSWorld-G, $+5.8$ ScreenSpot-V2) over the base. The same-size EMA mode lands within roughly one OOD-mean point of the cross-size $4$B variant ($65.2$ vs $66.3$) without an external teacher.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。