发现大模型安全对齐存在位置依赖性失效，提出针对性修复方法。
Rethinking Deep Alignment Through The Lens Of Incomplete Learning
- 通过分析自回归训练中的梯度衰减，揭示安全学习不完整的机制。
- 在多个模型上实现48%至98%的攻击成功率降低，且保持通用能力。
- 提出基模型偏好词作为指示器，可精准定位并修复薄弱区域。
大型语言模型尽管经过大量安全对齐训练，仍存在系统性漏洞易受对抗攻击。我们通过机制分析发现，自回归训练过程中位置相关的梯度减弱导致信号衰减，引发安全学习不完整——即安全训练未能充分改变模型在生成序列后半部分的偏好。为此，我们引入基模型偏好词（base-favored tokens），即基模型概率高于对齐模型的词汇元素，作为不完整安全学习的计算指标，并提出一种基于自适应惩罚与混合教师蒸馏的定向补全方法。在Llama和Qwen模型族上的实验表明，该方法显著提升对抗鲁棒性，攻击成功率下降48%至98%，同时保持模型通用能力。研究建立了安全对齐根本局限的机制理解，并提供实用解决方案。
原文摘要 · Abstract (English)
Large language models exhibit systematic vulnerabilities to adversarial attacks despite extensive safety alignment. We provide a mechanistic analysis revealing that position-dependent gradient weakening during autoregressive training creates signal decay, leading to incomplete safety learning where safety training fails to transform model preferences in later response regions fully. We introduce base-favored tokens -- vocabulary elements where base models assign higher probability than aligned models -- as computational indicators of incomplete safety learning and develop a targeted completion method that addresses undertrained regions through adaptive penalties and hybrid teacher distillation. Experimental evaluation across Llama and Qwen model families demonstrates dramatic improvements in adversarial robustness, with 48--98% reductions in attack success rates while preserving general capabilities. These results establish both a mechanistic understanding and practical solutions for fundamental limitations in safety alignment methodologies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。