通过宽度扩展实现大模型持续预训练,减少遗忘且保持原功能。
SCALE: Upscaled Continual Learning of Large Language Models
- 在不修改原始参数的前提下,通过轻量级扩展模块提升模型容量。
- 在韩语持续预训练中,英文任务遗忘率降低,韩语性能达最优平衡。
- 提出保留与协同适应双原则,稳定优化过程,适合长期增量学习场景。
我们重新审视大语言模型的持续预训练,认为当前进展更依赖于结构的合理扩展而非单纯增大参数量。提出SCALE架构,通过在线性模块中插入轻量级扩展并冻结预训练参数,保留残差与注意力拓扑结构,提升容量的同时不干扰原模型功能。该方法基于两个原则:持久保留(通过保留导向初始化和参数冻结维持基模型行为)与协同适应(选择性训练部分扩展组件以最小干扰获取新知识)。具体实现为SCALE-Preserve(优先保留)、SCALE-Adapt(优先适应)及可选的SCALE-Route路由扩展,实现令牌级在保留与适应头间的路由。在合成传记基准测试中,SCALE缓解了深度扩展导致的严重遗忘,同时仍能学习新知识;在韩语语料的持续预训练中,各变体在英语评估上遗忘更少,韩语基准表现优异,整体实现最佳稳定性与灵活性权衡。配套分析揭示了保留成立的条件,并说明保留与适应的交互如何稳定优化过程,优于标准持续学习设置。
原文摘要 · Abstract (English)
We revisit continual pre-training for large language models and argue that progress now depends more on scaling the right structure than on scaling parameters alone. We introduce SCALE, a width upscaling architecture that inserts lightweight expansion into linear modules while freezing all pre-trained parameters. This preserves the residual and attention topologies and increases capacity without perturbing the base model's original functionality. SCALE is guided by two principles: Persistent Preservation, which maintains the base model's behavior via preservation-oriented initialization and freezing of the pre-trained weights, and Collaborative Adaptation, which selectively trains a subset of expansion components to acquire new knowledge with minimal interference. We instantiate these ideas as SCALE-Preserve (preservation-first), SCALE-Adapt (adaptation-first), and SCALE-Route, an optional routing extension that performs token-level routing between preservation and adaptation heads. On a controlled synthetic biography benchmark, SCALE mitigates the severe forgetting observed with depth expansion while still acquiring new knowledge. In continual pre-training on a Korean corpus, SCALE variants achieve less forgetting on English evaluations and competitive gains on Korean benchmarks, with these variants offering the best overall stability-plasticity trade-off. Accompanying analysis clarifies when preservation provably holds and why the interplay between preservation and adaptation stabilizes optimization compared to standard continual learning setups.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。