提出联合微调框架，解决大模型训练中遗忘问题
Understanding Forgetting in LLM Supervised Fine-Tuning and Preference Learning -- A Convex Optimization Perspective
- 从凸优化视角证明分步训练存在遗忘缺陷
- 联合训练提升性能，最多提高23%且计算开销低
- 适合追求高效可靠微调的AI研发人员
大语言模型的后训练通常包含监督微调（SFT）和偏好学习（RLHF或DPO）两个阶段，当前主流采用依次执行的方式。然而，这种顺序范式在两者之间存在权衡：模型在第二阶段训练时会逐渐遗忘第一阶段的学习内容。尽管该方法因简单易扩展而被广泛使用，但其理论上的次优性长期未被解决。本文从凸优化角度理论上证明了顺序后训练的非最优性，并提出一种实用的联合后训练框架，具备理论收敛保证。实验表明，该框架在多个大模型评估基准上整体性能提升最高达23%，同时计算开销几乎不变。代码已开源于https://github.com/heshandevaka/XRIGHT。
原文摘要 · Abstract (English)
The post-training of LLMs, which typically consists of the supervised fine-tuning (SFT) stage and the preference learning stage (RLHF or DPO), is crucial to effective and safe LLM applications. The widely adopted approach in post-training popular open-source LLMs is to sequentially perform SFT and RLHF/DPO. However, this is suboptimal in terms of SFT and RLHF/DPO trade-off: the LLM gradually forgets about the first stage's training when undergoing the second stage's training. This sequential paradigm persists largely due to its simplicity and modularity, which make it easier to implement and manage at scale despite its limitations. We theoretically prove the sub-optimality of sequential post-training and propose a practical joint post-training framework which has theoretical convergence guarantees and empirically outperforms sequential post-training framework, with up to 23% overall performance improvement across multiple LLM evaluation benchmarks, while having minimal computational overhead. Our code is available at https://github.com/heshandevaka/XRIGHT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。