通过筛选高风险偏好数据,避免大模型对齐时遗忘预训练能力。
Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
- 基于三个数据特征构建风险评分,筛选有害偏好样本。
- 在不损失对齐效果前提下,显著保留模型基础能力。
- 适合关注模型稳定性与泛化性的研究者和开发者。
将大语言模型对齐人类偏好对实际部署至关重要,但常导致灾难性遗忘,使预训练通用能力退化。现有工作多将其视为优化或架构问题,而偏好数据本身如何引发退化仍缺乏研究。本文提出BALIGN,一种平衡的数据选择策略,在优化对齐效果的同时缓解灾难性遗忘。通过对偏好优化梯度的理论与实证分析,我们识别出三个关键数据特征:参考模型的对数概率差值、优选与劣选回复的词元长度差异,以及与通用能力语料库的TF-IDF相似度。将这些正交特征融合为综合风险评分,系统过滤可能破坏模型内在参数或提供低效对齐信号的高风险样本。在标准人类偏好数据集上的大量实验表明,BALIGN在不牺牲对齐效果的前提下,显著保持基础能力,始终达到最优帕累托前沿,且计算开销极小。
原文摘要 · Abstract (English)
Aligning large language models to human preferences is crucial for real-world deployment but frequently incurs an alignment tax, leading to the catastrophic forgetting of pre-trained general capabilities. While previous works primarily frame this problem as an optimization or architectural challenge, the inherent characteristics of preference data that drive this degradation remain largely underexplored. In this paper, we propose BALIGN, a balanced data selection strategy that explicitly mitigates catastrophic forgetting while optimizing alignment efficacy. Through theoretical and empirical analyses of the preference optimization gradient, we identify three key data-centric features that dictate parameter drift: the reference model's log-probability margin, the token length difference between chosen and rejected responses, and the TF-IDF similarity to general capability corpora. By aggregating these orthogonal features into a unified composite risk score, BALIGN systematically filters out high-risk preference samples that disrupt intrinsic model parameters or provide minimal alignment utility. Extensive experiments on standard human preference datasets demonstrate that BALIGN strongly preserves foundational capabilities without compromising alignment gains, consistently achieving the optimal Pareto frontier with minimal computational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。