无需重新标注，用新框架自动修正大模型因政策变化产生的错误对齐。
The Realignment Problem: When Right becomes Wrong in LLMs
- 基于冲突检测和双层优化，从已有数据中筛选需调整的样本。
- 在多个模型上验证有效，提升对齐精度且不损害通用能力。
- 适合需要持续更新对齐策略的AI系统研发团队使用。
大语言模型的后训练对齐依赖于随时间变化的政策规范，但文化变迁、价值重释及法规更新导致静态对齐日益脆弱。随着政策演进，已部署模型可能偏离当前对齐目标，形成难以审计和修正的对齐-现实差距。现有修复方法需重新人工标注，面临指南模糊、标注者理解漂移和大规模一致性下降等系统性挑战。本文提出TRACE（基于对齐冲突评估的分诊与再对齐），将再对齐转化为无需新增人工标注的结构化优化问题。利用更强模型作为代理裁判，通过三阶段流程：(1) 根据对齐冲突将偏好对分为倒置、抑制或保留三类；(2) 通过双层优化计算对齐影响得分，优先选择高影响力样本；(3) 采用混合目标函数，结合关系损失（如IPO）处理偏好倒置，惩罚性损失（如NPO）处理响应抑制。在Qwen2.5-7B、Gemma-2-9B和Llama-3.1-8B上的实验表明，该方法在合成基准和PKU-SafeRLHF数据集上实现稳健再对齐，且未降低通用性能。本工作为动态标注政策下的大模型再对齐提供了可扩展方案。代码已开源：https://respailab.github.io/TRACE/
原文摘要 · Abstract (English)
Post-training alignment of large language models (LLMs) relies on large-scale human annotations guided by policy specifications that change over time. Cultural shifts, value reinterpretations, and regulatory or industrial updates make static alignment increasingly brittle. As policies evolve, deployed models can diverge from current alignment objectives, creating an Alignment-Reality Gap that is difficult to audit or correct. Existing remediation typically requires re-annotation under revised guidelines, which introduces systematic challenges, including guideline ambiguity, annotator interpretation drift, and reduced consistency at scale. We introduce TRACE (Triage and Re-align by Alignment Conflict Evaluation), a framework that transforms realignment into a structured optimization problem over existing data without requiring fresh human annotation. Leveraging a stronger model as a proxy judge, TRACE operates via a three-stage pipeline: (1) triaging preference pairs into inversion, suppression, or retention categories based on alignment conflicts; (2) computing an alignment impact score via bi-level optimization to prioritize high-leverage samples; and (3) executing updates using a hybrid objective that combines relational losses (e.g., IPO) for preference inversion and punitive losses (e.g., NPO) for response suppression. Experiments on Qwen2.5-7B, Gemma-2-9B, and Llama-3.1-8B demonstrate robust realignment on synthetic benchmarks and the PKU-SafeRLHF dataset without degrading general utility. This work provides a scalable approach for LLM realignment under evolving data annotation policies and alignment guidelines. We release our code: https://respailab.github.io/TRACE/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。