发现视觉模型中间层存在安全漏洞，早退会增加有害输出风险。
Layer-wise Alignment: Examining Safety Alignment Across Image Encoder Layers in Vision Language Models
- 分析图像编码器各层安全对齐情况，发现有害信息分布不均。
- 早退图像编码器使三个模型生成有害内容概率显著上升。
- 提出分层强化学习算法L-PPO，有效缓解早退导致的安全问题。
视觉语言模型（VLMs）能力虽大幅提升，但其复杂架构使得安全对齐困难。本文揭示图像编码器中间层存在有害信息分布不均现象，并发现提前退出特定层会显著提高VLM生成有害响应的概率，称之为‘图像编码器早退漏洞’（ICET）。在LLaVA-1.5、LLaVA-NeXT和Llama 3.2三款模型上的实验表明，图像编码器早期退出显著提升有害输出概率。为此，我们提出一种改进的剪裁近端策略优化（Clip-PPO）算法，用于实现多模态分层强化学习对齐，称为层间PPO（L-PPO）。在三个多模态数据集上的评估显示，该方法能持续降低早退引发的有害性。
原文摘要 · Abstract (English)
Vision-language models (VLMs) have improved significantly in their capabilities, but their complex architecture makes their safety alignment challenging. In this paper, we reveal an uneven distribution of harmful information across the intermediate layers of the image encoder and show that skipping a certain set of layers and exiting early can increase the chance of the VLM generating harmful responses. We call it as "Image enCoder Early-exiT" based vulnerability (ICET). Our experiments across three VLMs: LLaVA-1.5, LLaVA-NeXT, and Llama 3.2, show that performing early exits from the image encoder significantly increases the likelihood of generating harmful outputs. To tackle this, we propose a simple yet effective modification of the Clipped-Proximal Policy Optimization (Clip-PPO) algorithm for performing layer-wise multi-modal RLHF for VLMs. We term this as Layer-Wise PPO (L-PPO). We evaluate our L-PPO algorithm across three multimodal datasets and show that it consistently reduces the harmfulness caused by early exits.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。