解决视觉位置编码失效导致的坐标预测偏差问题
Mitigating Coordinate Prediction Bias from Positional Encoding Failures
- 通过打乱位置编码，提取模型内在空间先验偏差
- 在ScreenSpot-Pro上实现各规模模型定位精度提升
- 无需重训练，在推理阶段即可修正坐标漂移
尽管多模态大模型在视觉-语言任务中表现优异，但在高分辨率输入下，视觉位置编码（VPE）性能下降，导致精确坐标预测仍具挑战。我们发现编码失败并非产生随机噪声，而是引发可预测的定向偏差，表明当定位信号弱时，模型会依赖内部空间先验。为此，我们提出一种无需训练、仅在推理阶段生效的纠正方法——视觉位置编码打乱引导（VPSG）。该方法通过打乱VPE，分离出与位置无关的倾向性，并利用这些负向证据，通过轻量级有限状态机引导数字解码。在ScreenSpot-Pro基准测试中，VPSG有效缓解了坐标漂移，且在不同规模模型上均带来稳定定位精度提升，无需任何再训练。代码已开源。
原文摘要 · Abstract (English)
While Multimodal Large Language Models (MLLMs) excel at general vision-language tasks, precise coordinate prediction remains a significant challenge, particularly as high-resolution inputs cause visual positional encodings (VPEs) to degrade. We demonstrate that these encoding failures do not result in random noise but instead trigger predictable, directional biases, suggesting that models default to internal spatial priors when grounding signals are weak. To counteract this, we introduce Vision-PE Shuffle Guidance (VPSG), a training-free, inference-time correction method. VPSG isolates position-unconditioned tendencies by shuffling VPEs and utilizes this negative evidence to steer digit decoding through a lightweight finite-state machine. Evaluation on the ScreenSpot-Pro benchmark confirms that VPSG effectively rectifies coordinate drift, yielding consistent improvements in localization accuracy across various model scales without any retraining. Our code is available at https://github.com/taoxj2001/VPSG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。