利用坐标数字位值特性,实现对GUI定位模型的高效攻击
MissClick: Exploiting Digit-Serialized Coordinates to Attack GUI Grounding Models

- 基于坐标数字的位值结构设计针对性攻击目标
- 无目标攻击成功率提升至75.07%,有目标攻击达62.67%
- 适用于安全测试与对抗鲁棒性研究
近期的GUI视觉定位模型将屏幕坐标以数字词元序列生成,经解析后映射为可执行点击。我们发现每个坐标数字作为分类词元预测,但百位数字变动1个单位会导致坐标偏移100单位,引发显著点击位移。这一现象揭示了现有攻击忽略数值结构的问题。因此,我们提出针对不同攻击目标的优化策略:无目标攻击(MissClick-U)最大化软坐标偏移,有目标攻击(MissClick-T)最小化带位权的目标数字损失。在OS-Atlas和UGround数据集上,跨桌面、网页和移动端平台的实验表明,MissClick-U分别取得75.07%和72.93%的成功率(较基线+16.62和+30.72个百分点),MissClick-T分别达到44.86%和62.67%(+31.73和+47.06个百分点)。结果验证了软坐标偏移在无目标攻击中更优,而位权目标数字优化在有目标攻击中表现最佳。
原文摘要 · Abstract (English)
Recent GUI visual grounding models generate screen coordinates as sequences of digit tokens that are parsed into numerical values and mapped to executable clicks. The security implications of this coordinate generation process have been largely overlooked. We observe that each coordinate digit is predicted as a categorical token, yet after parsing, changing a hundreds-place digit by one changes the corresponding numerical coordinate component by 100 units, which can induce a large displacement of the executed click. This observation motivates attack objectives that account for the numerical and place-value structure of coordinate outputs rather than treating them as ordinary text. Moreover, untargeted and targeted attacks impose different success conditions--displacing the click outside the correct region versus into an attacker-specified region--and therefore benefit from different objectives. We propose MissClick, a simple and effective white-box adversarial attack with two goal-specific objectives: MissClick-U maximizes soft-coordinate displacement for untargeted disruption, while MissClick-T minimizes a place-weighted target-digit loss for targeted hijacking. Compared with existing attacks against GUI grounding models on OS-Atlas and UGround across desktop, web, and mobile platforms, MissClick-U achieves untargeted success rates of 75.07\% and 72.93\% (+16.62 and +30.72 pp), and MissClick-T achieves targeted success rates of 44.86\% and 62.67\% (+31.73 and +47.06 pp). Attack objective comparison further shows that soft-coordinate displacement yields the highest untargeted attack success rate, whereas place-weighted target-digit optimization yields the highest targeted attack success rate, revealing distinct objective preferences for the two attack goals.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。