首次研究离散图像分词器的对抗鲁棒性，提出高效通用攻击与无监督防御方法。
On the Adversarial Robustness of Discrete Image Tokenizers
- 设计针对分词器特征的高效对抗攻击，可扰动提取的离散令牌。
- 无监督对抗微调显著提升对多种攻击的鲁棒性，且泛化能力强。
- 无需标签数据，适用于未见任务，适合构建安全多模态模型。
离散图像分词器将视觉输入编码为有限词汇表中的令牌序列，在多模态系统中日益流行，包括仅编码器、编码器-解码器和仅解码器模型。然而，与CLIP编码器不同，其对抗攻击脆弱性尚未被探索。本文是首个研究该问题的工作：首先提出旨在扰动分词器提取特征并改变输出令牌的攻击方法，这些攻击计算高效、与应用无关，在分类、多模态检索和图像描述任务中均有效。其次，受近期鲁棒CLIP编码器工作的启发，我们采用无监督对抗训练微调主流分词器，冻结其余组件。尽管无监督且任务无关，该方法显著提升了对无监督及端到端监督攻击的鲁棒性，并在未见任务和数据上具有良好泛化能力。相比有监督对抗训练，本方法可利用无标签图像，更具灵活性。总体而言，本工作凸显了分词器鲁棒性在下游任务中的关键作用，为构建安全多模态基础模型迈出重要一步。
原文摘要 · Abstract (English)
Discrete image tokenizers encode visual inputs as sequences of tokens from a finite vocabulary and are gaining popularity in multimodal systems, including encoder-only, encoder-decoder, and decoder-only models. However, unlike CLIP encoders, their vulnerability to adversarial attacks has not been explored. Ours being the first work studying this topic, we first formulate attacks that aim to perturb the features extracted by discrete tokenizers, and thus change the extracted tokens. These attacks are computationally efficient, application-agnostic, and effective across classification, multimodal retrieval, and captioning tasks. Second, to defend against this vulnerability, inspired by recent work on robust CLIP encoders, we fine-tune popular tokenizers with unsupervised adversarial training, keeping all other components frozen. While unsupervised and task-agnostic, our approach significantly improves robustness to both unsupervised and end-to-end supervised attacks and generalizes well to unseen tasks and data. Unlike supervised adversarial training, our approach can leverage unlabeled images, making it more versatile. Overall, our work highlights the critical role of tokenizer robustness in downstream tasks and presents an important step in the development of safe multimodal foundation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。