定位大模型对齐失效的训练源，提升安全性和可解释性。
TRACEALIGN -- Tracing the Drift: Attributing Alignment Failures to Training-Time Belief Sources in LLMs
- 通过语义冲突指数追踪生成内容与对齐策略的偏差来源。
- 在基准测试中降低85%对齐漂移，拒绝率提升且性能损失小于0.2。
- 适合研究模型安全、对齐机制或可解释性的研究人员使用。
大语言模型在微调以对齐人类价值观时，常因对抗性提示、解码扰动或改写越狱攻击导致输出不安全内容，即出现对齐漂移。现有工作多从行为层面描述失败现象，但缺乏对训练阶段信念来源的深入理解。本文提出TraceAlign，一个统一框架，可将不安全生成追溯至训练语料中的根源。核心是信念冲突指数（BCI），基于后缀数组匹配检索训练文档，量化生成片段与对齐政策间的语义不一致。提出三种互补干预：(i) TraceShield，推理时拒绝高BCI片段；(ii) 对比信念去混淆损失，微调中惩罚高BCI延续；(iii) Prov-Decode，基于出处感知的解码策略，规避可能产生高BCI扩展的分支。三者联合使对齐漂移降低达85%（在自建对齐漂移基准ADB上），标准任务性能下降小于0.2，拒绝质量提升。进一步通过后缀数组片段统计推导出漂移可能性的理论上限，揭示记忆频率与长度对对抗再激活风险的影响。TraceAlign首次提供可扩展、可追溯、基于证据的对齐失效分析与缓解工具包。代码已开源。
原文摘要 · Abstract (English)
Large Language Models (LLMs) fine-tuned to align with human values often exhibit alignment drift, producing unsafe or policy-violating completions when exposed to adversarial prompts, decoding perturbations, or paraphrased jailbreaks. While prior work has behaviorally characterized alignment failure, little is known about the training-time belief sources underlying these failures. We introduce TraceAlign, a unified framework for tracing unsafe completions back to their root causes in the model's training corpus. Central to our approach is the Belief Conflict Index (BCI), which quantifies semantic inconsistency between generated spans and aligned policies, based on retrieved training documents using suffix-array matching. We propose three complementary interventions: (i) TraceShield, an inference-time safety filter that refuses completions with high-BCI spans, (ii) Contrastive Belief Deconfliction Loss, a contrastive fine-tuning objective penalizing high-BCI continuations during DPO, and (iii) Prov-Decode, a provenance-aware decoding strategy that vetoes beam expansions predicted to yield high-BCI spans. Together, these defenses reduce alignment drift by up to 85% on our curated Alignment Drift Benchmark (ADB) while preserving utility on standard tasks, with delta less than 0.2 and improved refusal quality. We further derive a theoretical upper bound on drift likelihood via suffix-array span statistics, linking memorization frequency and length to adversarial reactivation risk. TraceAlign thus provides the first scalable, traceable, and grounded toolkit for understanding and mitigating alignment failures at source. To encourage further exploration and development, we open-source our implementation at: https://anonymous.4open.science/r/tracealign-2DA7
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。