对比五种方向防御在多模型上的表现,发现无一全能,需按架构单独校准。
A Cross-Architecture Audit of Direction-Based Inference-Time Defences in Vision-Language Models
- 在多个模型层中用控制大小的干扰方向测试防御效果
- 图像条件偏移在多数模型上保持性能损失接近测量噪声水平
- 不同模型架构间防御方向不通用,需针对语言解码器家族单独设计
针对视觉语言模型越狱攻击的推理时防御常通过在选定解码器层的残差流中减去一个校准方向实现。本文在四个架构家族的15个模型-层组合上,采用幅度受控协议,对五种防御候选方法进行对比:图像条件偏移均值、CMRM风格拒绝方向、ShiftDC风格攻击特定残差、提示指令忽略图像,以及同范数随机控制。结果显示,无单一候选在拒绝恢复与效用保留上均占优。图像条件偏移在LLaVA 1.5和Pixtral 12B上领先,且是唯一在所有架构中使效用损失保持在测量噪声基线的方法。提示指令在Qwen2.5 VL上表现最佳,攻击特定残差在Qwen2 VL 2B上最优。图像条件方向在15个单元中有13个表现出方向特异性,但高度依赖架构,仅在LLaVA 1.5 13B与Pixtral 12B这对兼容维度上无法迁移。此外,我们建立了纯文本与多模态拒绝几何的关联:CMRM方向与图像条件偏移在全部15个单元中具有正余弦对齐,均值0.35(范围0.17–0.65),较随机向量高15至25倍,符号检验p≈3e-5。结果表明,两种防御方案恢复部分重叠的几何结构,方向防御应为每类语言解码器架构分别校准。
原文摘要 · Abstract (English)
Inference time defences against vision language model jailbreaks often subtract a calibrated direction from the residual stream at a chosen decoder layer. We compare five defence candidates across 15 model and layer cells from four architectural families under a magnitude controlled protocol that matches the intervention size for each prompt and pairs every direction with a random control of the same norm. The candidates are the mean image conditioning shift, a CMRM style refusal direction, a ShiftDC style attack specific residual, a prompt instruction to ignore the image, and a random control. No single candidate dominates on both refusal recovery and utility preservation. The image conditioning shift leads on LLaVA 1.5 and Pixtral 12B and is the only candidate whose utility loss remains at the measurement noise floor in every family. The prompt instruction leads on Qwen2.5 VL, while the attack specific residual leads on Qwen2 VL 2B. The image conditioning direction is direction specific in 13 of 15 cells, but strongly architecture specific and nontransferable across the only dimension compatible pair, LLaVA 1.5 13B and Pixtral 12B. We also connect text only and multimodal refusal geometry. The CMRM direction has positive cosine alignment with the image conditioning shift in all 15 cells, with mean 0.35, range 0.17 to 0.65, 15 to 25 times the random vector null, and a sign test p value of about 3e-5. These results show that the two recipes recover partially overlapping geometry and that direction based defences should be calibrated separately for each language decoder family.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。