通过跨层低秩残差结构，实现高效大模型训练。
CR-Net: Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structure
- 利用跨层激活残差的低秩特性，设计双路径重建机制。
- 在60M到7B参数模型上均超越现有方法，内存与计算更少。
- 适合资源受限场景下的大模型高效预训练，如边缘设备。
低秩架构在高效大语言模型预训练中日益重要，显著降低参数量、内存和计算开销。然而现有方法存在三大缺陷：模型性能下降、计算开销大、激活内存节省有限。为此，我们提出跨层低秩残差网络（CR-Net），基于发现的层间激活残差具有低秩特性，采用双路径架构，通过结合前层输出与其低秩差值，高效重建层激活，以极少参数保留高秩信息。进一步设计专用于CR-Net的激活重计算策略，大幅降低内存需求。在60M至7B参数规模的模型上进行广泛预训练实验表明，CR-Net持续优于当前最先进低秩框架，且所需计算资源和内存更少。
原文摘要 · Abstract (English)
Low-rank architectures have become increasingly important for efficient large language model (LLM) pre-training, providing substantial reductions in both parameter complexity and memory/computational demands. Despite these advantages, current low-rank methods face three critical shortcomings: (1) compromised model performance, (2) considerable computational overhead, and (3) limited activation memory savings. To address these limitations, we propose Cross-layer Low-Rank residual Network (CR-Net), an innovative parameter-efficient framework inspired by our discovery that inter-layer activation residuals possess low-rank properties. CR-Net implements this insight through a dual-path architecture that efficiently reconstructs layer activations by combining previous-layer outputs with their low-rank differences, thereby maintaining high-rank information with minimal parameters. We further develop a specialized activation recomputation strategy tailored for CR-Net that dramatically reduces memory requirements. Extensive pre-training experiments across model scales from 60M to 7B parameters demonstrate that CR-Net consistently outperforms state-of-the-art low-rank frameworks while requiring fewer computational resources and less memory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。