连续微调不会放大模型缺陷,反而常导致混乱,只在特定强化学习场景下才可能加剧。
Iterative Finetuning is Mostly Idempotent

- 通过逐代用自身输出训练模型,测试行为倾向是否增强
- 非强化学习微调下倾向基本不变或减弱,仅少数情况会放大但损害连贯性
- 强化学习中若持续训练自身输出,倾向易放大,适合关注安全对齐的研究者
若模型存在某种行为倾向(如谄媚或错位),在基于自身输出进行连续微调时,该倾向是否会增强?我们通过一系列实验研究:每一代模型均在前一代生成的数据上微调,初始模型设定特定人格或信念。测试三种设置:指令模型上的监督微调(SFT)、基础模型上的合成文档微调(SDF)和直接偏好优化(DPO)。在SFT与SDF设置中,倾向大多衰减或保持不变,后续微调无明显效果;当出现放大时,通常以牺牲连贯性为代价。在DPO设置中,若持续以自身输出为偏好目标进行训练,倾向可稳定放大,但若每轮重新初始化,则放大现象消失。结果表明,倾向放大主要源于持续后训练,限制该阶段或可有效防御。对于非强化学习微调,倾向放大极为罕见且对数据量极度敏感,意外发生可能性很低。此外,放大与连贯性的权衡本身构成天然抑制机制。
原文摘要 · Abstract (English)
If a model has some behavioral tendency, such as sycophancy or misalignment, and it is trained on its own outputs, will the tendency be amplified in the next generation of models? We study this question by training a series of models where each model is finetuned on data generated by its predecessor, and the initial model is seeded with some persona or belief. We test three settings: supervised finetuning (SFT) on instruct models, synthetic document finetuning (SDF) on base models, and direct preference optimization (DPO). In the SFT and SDF settings, traits mostly decay or remain constant so that further finetuning cycles do nothing. In rare cases when amplification occurs, it generally comes at the cost of coherence. In the DPO setting, trait amplification can reliably occur when a model is continually trained with a preference for its own outputs, but vanishes when models are reinitialized at each cycle. Overall, our results suggest that amplification most likely comes from continual post-training, and limiting this stage may be an effective defense. For non-RL finetuning, trait amplification is rare and very sensitive to data quantity, making it significantly less likely to occur accidentally. Finally, the amplification-coherence tradeoff serves as a natural deterrent against trait amplification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。