用压缩谱分解训练大模型，让700亿参数模型能在掌上设备跑起来。
Spectral Compact Training: Pre-Training Large Language Models via Permanent Truncated SVD and Stiefel QR Retraction
- 用截断SVD压缩权重矩阵，训练时从不生成完整矩阵
- 在70B模型上实现199倍内存减少，掌上设备可全程训练
- 低秩下性能稳定，128秩为效率最优，适合资源受限场景
内存瓶颈仍是消费级硬件训练大语言模型的主要障碍。本文提出谱紧凑训练（Spectral Compact Training, SCT），将密集权重矩阵替换为截断SVD因子形式 W = U diag(s) V^T，训练与推理过程中始终不显式构造完整矩阵。梯度通过紧凑谱因子反向传播，每轮优化后使用QR分解将U、V投影回Stiefel流形。SCT在秩32时实现每MLP层最高199倍内存压缩，使70B参数模型可在Steam Deck手持设备上完成全量训练（峰值内存7.2 GB，对比密集FP32训练需1,245 GB）。在SmolLM2-1.7B上进行秩扫描实验（秩32-256，2000步，NVIDIA A100）显示，所有测试秩均收敛至相同损失底限（约4.2-4.5），表明学习率调度才是主要瓶颈而非MLP秩。秩128成为效率甜点，实现11.7倍压缩且困惑度最低。在秩32时，GPU内存下降46%，训练吞吐量翻倍。
原文摘要 · Abstract (English)
The memory wall remains the primary bottleneck for training large language models on consumer hardware. We introduce Spectral Compact Training (SCT), a method that replaces dense weight matrices with permanent truncated SVD factors W = U diag(s) V^T, where the full dense matrix is never materialized during training or inference. Gradients flow through the compact spectral factors via standard backpropagation, and U, V are retracted to the Stiefel manifold via QR decomposition after each optimizer step. SCT achieves up to 199x memory reduction per MLP layer at rank 32, enabling full training steps of 70B-parameter architectures on a Steam Deck handheld (7.2 GB peak memory vs. 1,245 GB for dense FP32 training with Adam). Rank-sweep experiments on SmolLM2-1.7B (ranks 32-256, 2000 steps, NVIDIA A100) show that all tested ranks converge to the same loss floor (~4.2-4.5), identifying the learning rate schedule -- not MLP rank -- as the primary bottleneck. Rank 128 emerges as the efficiency sweet spot at 11.7x MLP compression with the lowest perplexity. GPU memory drops 46% at rank 32 while training throughput doubles.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。