持续微调中模型会遗忘训练方法,而非内容。
Repeated post-training is not Self-improving: Diagnosing Scientific Amnesia in Continual DPO Pipelines
- 设计诊断工具识别模型在连续训练中丢失方法论知识的问题
- 实测5种策略,仅保守规则调度提升性能,其余均退化
- 揭示工业级持续微调存在方法遗忘,需谨慎评估优化方案
工业级大模型团队常通过多次对偏好数据集进行直接偏好优化(DPO)来迭代行为更新。这种模式的主导失败原因并非传统灾难性遗忘:模型可能保留已有行为,却无法积累可用于后续训练的方法论知识。我们称此为‘科学性遗忘’。本文将这一工程直觉转化为可测量的工业问题,提出:(i) 一套遗忘诊断工具,(ii) 基于程序的流水线,跨多个 Qwen2.5-7B-Instruct 的 FSDP 分片 DPO 检查点串联训练,(iii) 一个包含30轮的 HumanEval 子领域基准,(iv) 对五种策略提案者的对比诊断研究:随机记忆、基于规则的调度、仅检索记忆、冷启动贝叶斯优化,以及一种元科学记忆与推理候选方案 MSCL。在一个单种子、5条件 × 3步的真实模型链上,4/5策略在步骤级峰值 pass@1 上下降,包括 MSCL;只有人为保守的规则调度实现提升。后续小规模试点显示,在异构链中,仅 MSCL 完成并改善;在小规模多种子同质扫描中,仅检索记忆表现最优,且各候选间差异无统计显著性。因此,本文贡献在于诊断,而非宣称某方案解决该问题:科学性遗忘可在类生产级持续 DPO 流水线中观测到,对干预效果的结论高度依赖于链路架构、评估设计与种子覆盖范围。
原文摘要 · Abstract (English)
Industrial LLM teams often ship behavior updates by repeatedly DPO-training a base model on sequences of related preference-data campaigns. The dominant failure mode in this regime is not always classical catastrophic forgetting: a pipeline may preserve previously learned behaviors while still failing to accumulate reusable methodological knowledge about how to train the next campaign. We call this failure mode scientific amnesia. This paper turns that practitioner intuition into a measurable industrial problem. We contribute: (i) a diagnostic suite for amnesia, (ii) a Program-based pipeline that chains FSDP-sharded DPO checkpoints across Qwen2.5-7B-Instruct runs, (iii) a 30-campaign HumanEval subdomain benchmark, and (iv) a comparative diagnostic study of five strategy proposers: random memory, rule-based scheduling, retrieval-only memory, warm-start Bayesian optimization, and MSCL, a meta-scientific memory and reasoner candidate. Across a single-seed 5-condition * 3-step real-LM chain, 4 of 5 candidates degrade in step-level peak pass@1, including MSCL; only the deliberately conservative rule-based schedule improves. Follow-up pilots qualify rather than overturn this finding: in a heterogeneous chain, MSCL is the only completed candidate that improves, whereas in a small multi-seed homogeneous sweep, retrieval-only has the best mean Delta and no pairwise candidate gap is statistically distinguishable. The contribution is therefore diagnostic, not a claim that MSCL solves the problem: scientific amnesia is observable in a production-like continual-DPO pipeline, and conclusions about interventions depend sharply on chain regime, evaluator design, and seed coverage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。