通过权重缩放提升大模型训练效率与质量
WISCA: A Lightweight Model Transition Method to Improve LLM Training via Weight Scaling
- 提出WISCA方法，通过重缩放权重优化网络参数分布
- 在多架构上实现5.6%零样本性能提升，训练困惑度降低2.12%
- 适用于GQA架构和LoRA微调，无需改动模型结构
Transformer架构逐渐主导大语言模型（LLM）领域。近期针对基于Transformer的大语言模型训练优化主要集中在结构修改或优化器调整，但这些方法缺乏对训练过程中权重模式的系统性优化。权重模式指神经网络中权重参数的分布及其相对大小。为解决此问题，我们提出一种名为WISCA的权重缩放方法，通过有策略地改善神经网络权重模式，在不改变网络结构的前提下提升训练效率与模型质量。通过在保持模型输出不变的情况下重缩放权重，WISCA间接优化了模型的训练轨迹。实验表明，WISCA显著提升了收敛质量（以泛化能力与损失降低衡量），尤其在采用分组查询注意力（GQA）架构和LoRA微调任务中表现突出。实证结果表明，在多个架构上平均零样本验证性能提升5.6%，训练困惑度平均降低2.12%。
原文摘要 · Abstract (English)
Transformer architecture gradually dominates the LLM field. Recent advances in training optimization for Transformer-based large language models (LLMs) primarily focus on architectural modifications or optimizer adjustments. However, these approaches lack systematic optimization of weight patterns during training. Weight pattern refers to the distribution and relative magnitudes of weight parameters in a neural network. To address this issue, we propose a Weight Scaling method called WISCA to enhance training efficiency and model quality by strategically improving neural network weight patterns without changing network structures. By rescaling weights while preserving model outputs, WISCA indirectly optimizes the model's training trajectory. Experiments demonstrate that WISCA significantly improves convergence quality (measured by generalization capability and loss reduction), particularly in LLMs with Grouped Query Attention (GQA) architectures and LoRA fine-tuning tasks. Empirical results show 5.6% average improvement on zero-shot validation tasks and 2.12% average reduction in training perplexity across multiple architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。