视频大模型常漏掉明显有害内容,因采样和压缩设计缺陷。
Failures to Surface Harmful Contents in Video Large Language Models
- 发现主流视频大模型因帧采样稀疏、令牌压缩过度、视觉文本脱节,导致忽视明显有害内容。
- 在五款领先模型上测试,有害内容遗漏率超90%,即使全帧可见也常被忽略。
- 适合关注AI安全、视频生成与多模态模型漏洞的研究者阅读。
视频大语言模型(VideoLLMs)正广泛应用于各类关键场景,用户依赖自动摘要快速浏览视频内容。我们发现,这种交互方式隐藏了严重安全隐患:若视频中嵌入有害内容(如全屏插入或角落小块),尽管对人类观众清晰可见,当前顶尖的VideoLLMs却极少在输出中提及。根因分析揭示三个叠加的设计缺陷:(1)多数领先模型采用稀疏且均匀的帧采样,导致时间覆盖不足;(2)采样帧内存在激进的令牌降维,造成空间信息丢失;(3)编码器与解码器间连接断裂,生成文本时弱化视觉线索。基于此,我们构建了三种无需查询的黑盒攻击方法,精准匹配处理链中的缺陷。大规模评估显示,五款主流VideoLLMs中,有害内容遗漏率普遍超过90%。即使有害内容存在于所有帧中,模型仍持续失灵。这一结果凸显当前VideoLLMs设计的根本性漏洞,强调亟需在采样策略、令牌压缩与解码机制上确保语义覆盖,而非仅追求效率。
原文摘要 · Abstract (English)
Video Large Language Models (VideoLLMs) are increasingly deployed on numerous critical applications, where users rely on auto-generated summaries while casually skimming the video stream. We show that this interaction hides a critical safety gap: if harmful content is embedded in a video, either as full-frame inserts or as small corner patches, state-of-the-art VideoLLMs rarely mention the harmful content in the output, despite its clear visibility to human viewers. A root-cause analysis reveals three compounding design flaws: (1) insufficient temporal coverage resulting from the sparse, uniformly spaced frame sampling used by most leading VideoLLMs, (2) spatial information loss introduced by aggressive token downsampling within sampled frames, and (3) encoder-decoder disconnection, whereby visual cues are only weakly utilized during text generation. Leveraging these insights, we craft three zero-query black-box attacks, aligning with these flaws in the processing pipeline. Our large-scale evaluation across five leading VideoLLMs shows that the harmfulness omission rate exceeds 90% in most cases. Even when harmful content is clearly present in all frames, these models consistently fail to identify it. These results underscore a fundamental vulnerability in current VideoLLMs' designs and highlight the urgent need for sampling strategies, token compression, and decoding mechanisms that guarantee semantic coverage rather than speed alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。