通过视觉提示泄露漏洞，逆向推导出视觉语言模型的图像分块大小和预处理参数。
Steal the Patch Size: Adversarially Manipulate Vision-Language Models
- 利用视觉变换器的分块机制，通过网格图像测试识别分块尺寸
- 在多种开源与闭源模型上成功恢复分块大小和预处理分辨率
- 可被用于针对性攻击，适合安全研究与模型防护方向读者
我们提出一种黑盒模型窃取攻击，可恢复部署的视觉-语言模型（VLM）中私有的视觉分块配置，包括视觉分块大小和输入预处理流程。核心思路是利用基于ViT的分块机制产生的任务级侧信道：当合成网格图像与隐藏的分块网格对齐时，边界信息在分块阶段被抹除，导致分类准确率周期性下降。通过扫描不同网格单元尺寸并测量这些准确率塌陷，可推断出分块大小；再通过引入填充和一致性检测，进一步判断预处理是动态还是固定分辨率，并恢复目标重采样分辨率。我们在多个开源Qwen-VL变体及闭源模型（如GPT、Claude）上均可靠地恢复了分块相关参数。最后，我们证明此类泄露可支持预处理感知的迁移攻击和模型定向对抗操纵。
原文摘要 · Abstract (English)
We present a black-box model-stealing attack that recovers private vision-tokenizer configurations of deployed vision-language models (VLMs), including the visual patch size and input preprocessing pipeline. The key idea is a task-level side channel induced by ViT-style patchification: when a synthetic grid image is aligned with the hidden patch grid, boundary cues are erased at tokenization, causing periodic accuracy drop. By sweeping the grid cell size and measuring these collapses, we infer the patch size; by introducing padding and a consistency-check test, we further identify whether preprocessing is dynamic- or fixed-resolution and recover the target resize resolution. Across open-source Qwen-VL variants and proprietary models including GPT and Claude, we reliably recover tokenizer-related parameters. Finally, we show that such leakage enables preprocessing-aware transfer attacks and model-targeted adversarial manipulation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。