通过相似性正则化提升大模型预训练效果
SimReg: Achieving Higher Performance in the Pretraining via Embedding Similarity Regularization
- 在序列内对同标签词向量加强相似性，异标签间施加对比分离
- 训练收敛速度提升30%以上，零样本性能平均提高1%
- 适用于密集和MoE架构，适合追求高效预训练的研究者
采用下一词预测进行大规模语言模型（LLM）预训练虽取得显著进展，但此类模型中词元嵌入具有上下文依赖性，导致类内方差高、类间相似度大，阻碍表示学习效率。尽管基于相似性的正则化在监督微调与分类任务中已证明有效，其在大规模LLM预训练中的应用与效果仍不明确。本文提出SimReg，一种嵌入相似性正则化损失，显式鼓励序列内相同真实标签的词元表示更相似，同时通过对比损失使不同标签词元相互分离。分析表明，该机制通过扩大多分类边界实现性能提升，从而促进更高效的分类。在密集和混合专家（MoE）架构上的大量实验显示，SimReg可使训练收敛速度提升超30%，并在标准基准上平均提升零样本下游性能超1%。进一步的消融研究与分析为超参数调优和损失有效性提供了实用洞见。
原文摘要 · Abstract (English)
Pretraining large language models (LLMs) with next-token prediction has led to remarkable advances, yet the context-dependent nature of token embeddings in such models results in high intra-class variance and inter-class similarity, thus hindering the efficiency of representation learning. While similarity-based regularization has demonstrated benefit in supervised fine-tuning and classification tasks, its application and efficacy in large-scale LLM pretraining remains underexplored. In this work, we propose the SimReg, an embedding similarity regularization loss that explicitly encourages token representations with the same ground-truth label within each sequence to be more similar, while enforcing separation from different-label tokens via a contrastive loss. Our analysis reveals that this mechanism introduces gains by enlarging multi-classification margins, thereby enabling more efficient classification. Extensive experiments across dense and Mixture-of-Experts (MoE) architectures demonstrate that SimReg consistently accelerates training convergence by over 30% and improves average zero-shot downstream performance by over 1% across standard benchmarks. Further ablation studies and analyses offer practical insights into hyperparameter tuning and loss effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。