零推理成本提升低秩预训练效果，让小模型也能逼近大模型性能。
DLR: Zero-Inference-Cost Latent Residuals for Low-Rank Pre-Training
- 通过固定结构残差增强低秩权重，训练时无需新增参数。
- 在60M到7B参数的LLaMA模型上，困惑度显著降低，130M以上提升更明显。
- 部署时可完全融合，不影响推理性能，适合资源受限场景使用。
大规模语言模型推动了自然语言与多模态AI的发展，但其规模化预训练成本高昂。低秩预训练通过将权重矩阵分解为低秩乘积，减少参数量和计算量，是可行方案，但通常性能落后于全秩训练。本文提出仅用于训练的、无参数的可折叠插件DLR（Duplicated Latent Residual）。DLR在标准低秩输出Bz基础上，添加一个固定结构残差：alpha/sqrt(K) * Expand_K(z)，将每个隐状态坐标复制K = ceil(d_out/r)次至输出层。alpha固定后，每层不引入可学习参数；训练完成后，该残差可闭式融合至上投影矩阵中，即B* = B + alpha/sqrt(K) R^T，使部署后的参数量、浮点运算量和内存占用与原始低秩主干完全一致。在从60M到7B参数的LLaMA系列模型上，DLR在多数设置下提升了低秩预训练在C4验证集上的困惑度，尤其在130M及以上规模效果显著；融合后的检查点可无缝迁移至标准基准的监督微调任务。
原文摘要 · Abstract (English)
Large language models have driven recent progress in language and multimodal AI, yet pre-training them at scale is prohibitively expensive. Low-rank pre-training, which factorizes each weight matrix into a rank-r product to reduce both parameters and FLOPs, is a promising response but typically lags full-rank training in quality. We propose Duplicated Latent Residual (DLR), a training-only, parameter-free, foldable plug-in for low-rank pre-training. DLR augments the standard low-rank output Bz with a fixed structured residual alpha/sqrt(K) * Expand_K(z) that replicates each latent coordinate K = ceil(d_out/r) times across the output. With alpha fixed, DLR adds zero learnable parameters per layer; after training, it is absorbed into the up-projection in closed form, B* = B + alpha/sqrt(K) R^T, so deployment parameter count, FLOPs and memory match the underlying low-rank backbone exactly. Across LLaMA models from 60M to 7B parameters, DLR strengthens low-rank pre-training on C4 validation perplexity in most settings, with the clearest gains at 130M and above; folded checkpoints transfer cleanly to supervised fine-tuning on standard benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。