通过智能扩展关键参数，用更少参数实现更强语言模型
Advantageous Parameter Expansion Training Makes Better Large Language Models
- 识别并逐步扩展对性能关键的优越参数
- 指令微调仅用52%参数超越全参数训练
- 持续预训练仅需30%数据达同等效果
尽管增加可训练参数数量能提升大语言模型性能，但也会带来计算开销。我们发现，部分称为‘优越参数’的子集对模型表现起决定性作用，且更强模型拥有更多此类参数。本文提出优势参数扩展训练（APEX），通过将优越参数逐步扩展至劣势参数空间，在总参数量不变的前提下提高其比例，从而增强训练效率。在五个基础模型上的指令微调与持续预训练实验表明：指令微调中，APEX仅使用52%可训练参数即超越全参数微调；持续预训练中，仅需约30%训练数据即可达到传统训练的困惑度水平，并在下游任务上取得显著提升。
原文摘要 · Abstract (English)
Although scaling up the number of trainable parameters can effectively improve the training performance of large language models, it also leads to increased computational overhead. When delving into the parameter difference, we find that a subset of parameters, termed advantageous parameters, plays a crucial role in determining model performance. Further analysis reveals that stronger models tend to possess more such parameters. In this paper, we propose Advantageous Parameter EXpansion Training (APEX), a method that progressively expands advantageous parameters into the space of disadvantageous ones, thereby increasing their proportion and enhancing training effectiveness, while keeping the total parameter count unchanged. Extensive experiments on both instruction tuning and continued pre-training across five base models demonstrate that, in instruction tuning, APEX outperforms full-parameter tuning while using only 52% of the trainable parameters. In continued pre-training, APEX achieves the same perplexity level as conventional training with only approximately 30% of the training data, and yields significant improvements on downstream tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。