通过动态降秩与正交约束，实现高效低参数大模型预训练。
Efficient Pre-Training of LLMs through Truncated SVD Layers
- 基于谱能量启发式动态调整秩，避免固定秩的性能损失。
- 在多种模型规模下，性能媲美全参数模型，计算量显著降低。
- 适合追求高效高精度大模型训练的研究者与工业应用。
大规模语言模型（LLMs）的持续扩展使得预训练成本日益高昂。尽管低秩表示和正交权重矩阵理论上可减少参数量与计算开销，但现有方法多依赖静态秩选择且因计算成本高而难以强制权重正交。本文提出TSVD框架，在训练过程中始终维持低秩与严格正交性。其采用基于谱能量的启发式方法实现自适应秩选择，并引入缓存机制保障正交性。理论分析证明该方法在预训练动态中的优势，跨多种模型规模的实验表明其具有强有效性。TSVD在保持或超越全参数基线性能的同时，大幅降低计算需求，为高效高性能的大模型预训练提供了可靠、实用且可扩展的路径。
原文摘要 · Abstract (English)
The massive scaling of Large Language Models (LLMs) has made pretraining increasingly cost-prohibitive. While low-rank representation and orthonormal weight matrices could in principle reduce parameter counts and computational overhead, most existing methods rely on static rank selection and do not enforce weight orthonormality due to high computational cost. This paper introduces TSVD, a framework that maintains low rank and strict orthonormality throughout the training process. It utilizes a spectral energy-based heuristic for adaptive rank selection, and a caching mechanisms to maintain orthonormality. Theoretical analysis justifies the advantage of the approach in pretraining dynamics and experiments across various model scales demonstrate that it is effective empirically. TSVD matches or exceeds the performance of full-parameter baselines while significantly reducing compute requirements. The approach thus offers a well-founded, practical, and scalable path toward efficient high-performance LLM pretraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。