arXiv:2608.24354cs.CRcs.AI2026-08

修复多模态大模型后门,不依赖触发器信息即可有效消除风险

Not All Tokens Are Equal: Region-Aware Consistency Repair of Backdoors in MLLMs

论文配图:Not All Tokens Are Equal: Region-Aware Consistency Repair of Backdoors in MLLMs
图 1 · 摘自论文原文
  • 通过分析层间表征异常,定位触发特征所在区域并针对性修复
  • 在36种攻击设置下平均攻击成功率降至1.1%,32种情况下归零
  • 仅需100个干净样本,无需知道触发器或攻击目标

多模态大模型在用户应用中日益普及,但其构建流程继承了后门风险:触发器可能存在于图像、文本或两者中。现有模型级后门移除方法主要针对传统分类器,对多模态大模型效果有限;而多模态大模型特有防御多在推理时过滤可疑输入,无法清除模型中的后门。为填补这一空白并从源头消除多模态大模型的潜在后门,本文提出RACER,一种基于关键观察的模型级修复框架:后门会引发内部表征在层间演进异常,称为层间不一致异常。该异常具有模态依赖性,主要集中于后门模型实际依赖的触发特征所对应的令牌区域。RACER因此将融合表示分解为视觉与文本令牌区域,分别对层间不一致进行归一化,并在深层窗口内使用模态感知权重重新组合,形成区域感知的不一致目标,更准确捕捉局部后门引起的异常。通过最小-最大优化,该目标驱动最坏情况扰动合成及对抗微调,抑制后门行为所依赖的深层表征方向偏移。RACER仅需100个干净样本,无需触发器、攻击目标甚至是否含后门的知识。在三个开源多模态大模型上,跨36种包含图像、文本和多模态触发器的后门设置评估显示,平均攻击成功率降至1.1%,32种情况下为0%,同时保持干净任务性能。

原文摘要 · Abstract (English)

MLLMs are increasingly deployed in user-facing applications, yet they inherit backdoor risks from the pipelines used to construct them: triggers may reside in images, texts, or both. Existing model-level backdoor removal methods, largely designed for conventional classifiers, show limited effectiveness on MLLMs, while MLLM-specific defenses mainly operate at inference time, filtering suspicious inputs without removing the backdoor embedded in the model. To address this gap and eliminate latent backdoors from MLLMs at their source, we present RACER, a model-level repair framework motivated by a key observation: backdoors induce abnormal layer-to-layer evolution in internal representations, which we term the layer-wise inconsistency anomaly. Importantly, this anomaly is modality-dependent, concentrating primarily in the token region encoding the trigger features that the backdoor model actually relies on. RACER therefore decomposes the fused representation into visual and textual token regions, normalizes their layer-wise inconsistency separately, and recomposes them using modality-aware weights over a deep-layer window, yielding a region-aware inconsistency objective that better captures localized backdoor-induced anomalies. Through a min-max optimization, this objective drives worst-case perturbation synthesis and adversarial fine-tuning against the resulting perturbation to repair the model, suppressing the deep representational directional shifts on which backdoor behaviors rely. RACER requires only 100 clean samples and no knowledge of the trigger, attack objective, or even whether the input model contains a backdoor. Evaluations on three open-source MLLMs across 36 backdoor settings spanning image, text, and multimodal triggers show that RACER reduces the average ASR to 1.1%, reaching 0% in 32 settings, while preserving clean-task utility on both backdoor and clean models.

后门防御多模态模型模型修复

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。