让大模型安全行为不受性格设定影响,提升一致性。
Making Your LLMs More Objective: Stabilizing LLM Safety Behavior Across Traits with Trait-Invariant Safety Tuning
- 通过自蒸馏框架让模型在不同性格设定下保持一致的安全响应
- 实验显示新方法使危险请求拒绝率提升12.3%,且不降低通用能力
- 适合关注大模型安全稳定性的研究者与应用开发者
对齐的大语言模型应根据用户请求内容决定是否拒绝危险请求或遵守安全请求。然而我们发现,同一请求在不同系统设定的性格特征下可能产生截然不同的安全决策,这种现象称为‘性格诱导的安全波动’。为此,我们提出基于拒绝行为的度量指标:‘性格诱导偏离度’衡量数据集层面的偏差,‘性格诱导翻转率’衡量相同请求在不同性格下的决策差异。进一步分析发现,性格特征会扰动模型在低维空间中的安全表征。为实现性格不变的安全性,我们提出特质不变安全微调(TIST),一种简单有效的自蒸馏框架,将有性格条件的输出对齐无性格基准。在此基础上,我们设计了仅在识别出的特质子空间中施加不变性的实例——特质子空间中和(TraSN)。实验表明,TraSN在提升性格不变安全性的同时,增强对有害请求的拒绝能力,且保留模型整体性能。结果强调了性格设定是影响大模型安全性和鲁棒性的关键因素。
原文摘要 · Abstract (English)
Aligned large language models (LLMs) are expected to exhibit safety behavior based on the content of the user request: they should refuse unsafe requests and comply with safe ones. However, we show that the same request can elicit substantially different safety decisions under different traits assigned in the system prompt, a failure mode we call trait-induced safety variation. To measure this failure, we introduce refusal-based metrics: Trait-Induced Deviation measures dataset-level deviation from the no-trait baseline, while Trait-Induced Flip Rate measures whether the same request receives different safety decisions across traits. We then provide a representation-level analysis of the mechanism behind trait-induced safety shifts and find that traits perturb the model's safety representations within a low-dimensional subspace. To achieve trait-invariant safety, where safety behavior remains stable across traits, we introduce Trait-Invariant Safety Tuning (TIST), a simple yet effective self-distillation framework that aligns an LLM's trait-conditioned behavior with its no-trait behavior. Guided by our analysis, we further propose Trait-Subspace Neutralization (TraSN), an instantiation of TIST, which enforces invariance only within the identified trait subspace. Experiments show that TraSN improves trait-invariant safety and strengthens harmful-request safety while preserving general capability. Our results highlight traits as an important factor in LLM safety and robust model behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。