安全对齐会压制视觉模型的回答意愿,即使证据仍在。
When Safety Overrides Vision: Exploring Dynamics between Vision Influence and Safety Alignment in Vision-Language Models

- 分析视觉语言模型在安全约束下的内部解码过程。
- 拒绝回答时仍受视觉信息影响,感知基础未丢失。
- 干预特定激活可恢复正确回答,无需重训练。
对齐的视觉语言模型(VLMs)旨在平衡基于视觉的推理与安全生成行为。然而我们观察到一个显著现象:在安全约束指令下,模型常拒绝回答在默认指令下本可正确回答的问题,尽管图像-问题输入完全相同。这引发根本疑问:安全对齐是否抑制了感知基础本身,还是视觉证据仍内存在但生成被引导至回避?本文研究对齐VLM中安全诱导回避的内部解码动态。在多个架构和多模态基准上,我们发现拒绝生成在整个解码过程中仍持续受视觉证据影响,表明感知基础基本保留。进一步显示,尽管拒绝的表征组织因架构而异,安全指令始终改变后期隐藏状态动态,导向拒绝解码。通过针对性激活干预,抑制拒绝相关表征可可靠恢复接地回答行为,且无需重训练或修改视觉输入。这些发现揭示了对齐VLM中此前未被充分关注的失效模式:安全对齐可能在感知证据仍存在的前提下,覆盖视觉表达。
原文摘要 · Abstract (English)
Aligned vision-language models (VLMs) are designed to balance grounded visual reasoning with safe generation behavior. However, we observe a striking phenomenon: under safety-constrained instruction, models frequently abstain from answering questions that remain correctly answerable under default instruction despite receiving identical image-question inputs. This raises a fundamental question: does safety alignment suppress perceptual grounding itself, or does visual evidence remain internally available while generation is redirected toward abstention? In this work, we investigate the internal decoding dynamics underlying safety-induced abstention in aligned VLMs. Across multiple architectures and multimodal benchmarks, we show that abstained generations remain consistently influenced by visual evidence throughout decoding, indicating that perceptual grounding is largely preserved despite refusal behavior. We further demonstrate that, although the representational organization of refusal differs substantially across architectures, safety-constrained instruction consistently alters late-stage hidden-state dynamics toward refusal-oriented decoding. Finally, through targeted activation-level interventions, we show that suppressing refusal-related representations reliably restores grounded answering behavior across models without retraining or modifying visual inputs. Together, these findings reveal a previously underexplored failure mode in aligned VLMs: safety alignment can override grounded visual expression even when perceptual evidence remains internally preserved.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。