不微调模型就能让多模态大模型更安全，自动识别并阻断有害输出。
Automating Steering for Safe Multimodal Large Language Models
- 用安全感知分数自动定位模型中危险信号的关键层
- 在推理阶段检测到高风险时，通过轻量干预模块拒绝生成有害内容
- 适用于需要安全防护的多模态应用，如内容审核与对话系统
多模态大语言模型（MLLMs）虽具备强大的跨模态推理能力，但面对对抗性多模态输入时也带来新的安全风险。为提升推理阶段的安全性，我们提出无需微调底层模型的模块化自适应干预技术 AutoSteer。该技术包含三个核心组件：(1) 新提出的安全意识评分（SAS），可自动识别模型内部层中最具安全相关性的差异；(2) 一个训练用于估计中间表示产生有毒输出概率的自适应安全探测器；(3) 一个轻量级拒绝头，在检测到安全风险时选择性干预生成过程。在 LLaVA-OV 与 Chameleon 上的实验表明，AutoSteer 显著降低了文本、视觉及跨模态威胁的攻击成功率（ASR），同时保持了模型的通用能力。结果表明，AutoSteer 是一种实用、可解释且有效的多模态AI安全部署框架。
原文摘要 · Abstract (English)
Recent progress in Multimodal Large Language Models (MLLMs) has unlocked powerful cross-modal reasoning abilities, but also raised new safety concerns, particularly when faced with adversarial multimodal inputs. To improve the safety of MLLMs during inference, we introduce a modular and adaptive inference-time intervention technology, AutoSteer, without requiring any fine-tuning of the underlying model. AutoSteer incorporates three core components: (1) a novel Safety Awareness Score (SAS) that automatically identifies the most safety-relevant distinctions among the model's internal layers; (2) an adaptive safety prober trained to estimate the likelihood of toxic outputs from intermediate representations; and (3) a lightweight Refusal Head that selectively intervenes to modulate generation when safety risks are detected. Experiments on LLaVA-OV and Chameleon across diverse safety-critical benchmarks demonstrate that AutoSteer significantly reduces the Attack Success Rate (ASR) for textual, visual, and cross-modal threats, while maintaining general abilities. These findings position AutoSteer as a practical, interpretable, and effective framework for safer deployment of multimodal AI systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。