提出首个音频语言模型安全防御框架，有效拒绝有害请求且不误伤正常语音。
SARSteer: Safeguarding Large Audio-Language Models via Safe-Ablated Refusal Steering
- 用文本引导的拒绝策略控制音频输入，避开音视频激活差异问题。
- 通过分解安全空间抑制过度拒绝，保持对正常语音的响应能力。
- 适合关注音频大模型安全性的研究者与产品开发者使用。
大型音频语言模型（LALMs）正成为现实应用中关键的多模态基础架构。然而，最新研究表明，相比文本输入，音频输入更容易引发有害响应，带来新的部署风险。尽管安全对齐在大语言模型（LLMs）和大视觉语言模型（LVLMs）中已取得初步进展，但将这些方法直接应用于LALMs面临两大挑战：1）基于文本的引导策略在音频输入下失效，因激活分布差异过大；2）提示式防御会导致对良性语音查询产生过度拒绝。为此，我们提出安全消融拒绝引导（SARSteer），首个面向LALMs的推理时防御框架。SARSteer利用文本衍生的拒绝引导实现拒绝行为，避免修改音频输入，并引入分解式安全空间消融机制缓解过度拒绝。大量实验表明，SARSteer显著提升了有害查询的拒绝率，同时保留了对良性查询的响应能力，为LALMs的安全对齐提供了原则性方案。代码与构建的数据集已开源于 https://github.com/linweiii/SARSteer。
原文摘要 · Abstract (English)
Large Audio-Language Models (LALMs) are becoming essential as a powerful multimodal backbone for real-world applications. However, recent studies show that audio inputs can more easily elicit harmful responses than text, exposing new risks toward deployment. While safety alignment has made initial advances in LLMs and Large Vision-Language Models (LVLMs), we find that vanilla adaptation of these approaches to LALMs faces two key limitations: 1) LLM-based steering fails under audio input due to the large distributional gap between activations, and 2) prompt-based defenses induce over-refusals on benign-speech queries. To address these challenges, we propose Safe-Ablated Refusal Steering (SARSteer), the first inference-time defense framework for LALMs. Specifically, SARSteer leverages text-derived refusal steering to enforce rejection without manipulating audio inputs and introduces decomposed safe-space ablation to mitigate over-refusal. Extensive experiments demonstrate that SARSteer significantly improves harmful-query refusal while preserving benign responses, establishing a principled step toward safety alignment in LALMs. The codes and constructed datasets are released at https://github.com/linweiii/SARSteer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。