给文档截图加干扰像素，能轻易骗过视觉语言检索模型。
Document Screenshot Retrievers are Vulnerable to Pixel Poisoning Attacks
- 用像素级恶意干扰攻击视觉文档检索模型。
- 单张恶意截图让41.9%查询结果被污染。
- 对已知查询可实现100%攻击成功率，适合安全测试。
近期密集检索技术发展催生了基于视觉语言模型（VLM）的文档检索器，如DSE和ColPali，它们将文档截图嵌入向量空间以实现高效搜索，相比传统纯文本方法流程更简化。本文提出三种像素污染攻击方法，评估其在不同设置和参数下的有效性。实验表明，仅向检索库中注入一张对抗性截图，即可显著破坏搜索结果：在DSE上使41.9%的查询前10条结果被污染，在ColPali上为26.4%，该脆弱性远超同类文本检索器的攻击效果。当针对特定已知查询时，攻击成功率更高，部分情况下可达100%。本研究揭示了视觉语言模型在实际部署中的潜在风险。
原文摘要 · Abstract (English)
Recent advancements in dense retrieval have introduced vision-language model (VLM)-based retrievers, such as DSE and ColPali, which leverage document screenshots embedded as vectors to enable effective search and offer a simplified pipeline over traditional text-only methods. In this study, we propose three pixel poisoning attack methods designed to compromise VLM-based retrievers and evaluate their effectiveness under various attack settings and parameter configurations. Our empirical results demonstrate that injecting even a single adversarial screenshot into the retrieval corpus can significantly disrupt search results, poisoning the top-10 retrieved documents for 41.9% of queries in the case of DSE and 26.4% for ColPali. These vulnerability rates notably exceed those observed with equivalent attacks on text-only retrievers. Moreover, when targeting a small set of known queries, the attack success rate raises, achieving complete success in certain cases. By exposing the vulnerabilities inherent in vision-language models, this work highlights the potential risks associated with their deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。