arXiv:2511.12735cs.CV2025-11AAAI被引 1

通过多模态提示调优,实现对开放词汇目标检测器的隐蔽后门攻击。

Backdoor Attacks on Open Vocabulary Object Detectors via Multi-Modal Prompt Tuning

  • 联合优化图像与文本模态的可学习提示词,植入后门。
  • 在多个数据集上实现超90%的攻击成功率,且小触发器即可激活。
  • 适用于研究模型安全性的研究人员,尤其关注AI鲁棒性者。

开放词汇目标检测器(OVODs)融合视觉与语言能力,基于文本提示检测任意物体类别,具备强大的零样本泛化能力。随着这类模型在机器人、自动驾驶和监控等高风险场景中广泛应用,其安全风险亟需重视。本文首次研究了针对OVODs的后门攻击,揭示了提示调优引入的新攻击面。我们提出TrAP(Trigger-Aware Prompt tuning)——一种多模态后门注入策略,联合优化图像与文本模态中的提示参数及视觉触发器。TrAP通过轻量级可学习提示词植入恶意行为,无需重训练基础模型权重,既保留通用性能又嵌入隐藏后门。采用课程式训练策略逐步缩小触发器尺寸,使推理阶段仅用小型触发器即可有效激活攻击。在多个数据集上的实验表明,TrAP在目标误分类和目标消失攻击中均达到超过90%的成功率,同时相比零样本设置提升了下游数据集的干净图像表现。代码已公开:https://github.com/rajankita/TrAP

原文摘要 · Abstract (English)

Open-vocabulary object detectors (OVODs) unify vision and language to detect arbitrary object categories based on text prompts, enabling strong zero-shot generalization to novel concepts. As these models gain traction in high-stakes applications such as robotics, autonomous driving, and surveillance, understanding their security risks becomes crucial. In this work, we conduct the first study of backdoor attacks on OVODs and reveal a new attack surface introduced by prompt tuning. We propose TrAP (Trigger-Aware Prompt tuning), a multi-modal backdoor injection strategy that jointly optimizes prompt parameters in both image and text modalities along with visual triggers. TrAP enables the attacker to implant malicious behavior using lightweight, learnable prompt tokens without retraining the base model weights, thus preserving generalization while embedding a hidden backdoor. We adopt a curriculum-based training strategy that progressively shrinks the trigger size, enabling effective backdoor activation using small trigger patches at inference. Experiments across multiple datasets show that TrAP achieves high attack success rates for both object misclassification and object disappearance attacks, while also improving clean image performance on downstream datasets compared to the zero-shot setting. Code: https://github.com/rajankita/TrAP

后门攻击目标检测多模态提示调优

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。