训练时压缩大模型，60%-90%省内存还不丢性能
BackSlash: Rate Constrained Optimized Training of Large Language Models
- 训练中引入率失真优化，动态控制模型复杂度
- 压缩率高达90%且不降精度，优于训练后压缩
- 适合边缘设备部署，提升模型鲁棒性和泛化能力
大型语言模型的快速发展推动了训练后参数压缩的研究，但训练阶段的压缩仍鲜有探索。本文提出率约束训练（BackSlash），基于率失真优化（RDO）实现训练时压缩。该方法可在模型精度与复杂度间灵活权衡，显著降低参数冗余并保持性能。在多种架构与任务上的实验表明，BackSlash 可在无精度损失的前提下将内存使用减少60%至90%，相比训练后压缩具有显著优势。此外，该方法具备高度通用性：使用小拉格朗日乘子可增强泛化能力；对剪枝具有强鲁棒性，在80%剪枝率下仍保持精度；适用于边缘设备加速推理的网络简化。
原文摘要 · Abstract (English)
The rapid advancement of large-language models (LLMs) has driven extensive research into parameter compression after training has been completed, yet compression during the training phase remains largely unexplored. In this work, we introduce Rate-Constrained Training (BackSlash), a novel training-time compression approach based on rate-distortion optimization (RDO). BackSlash enables a flexible trade-off between model accuracy and complexity, significantly reducing parameter redundancy while preserving performance. Experiments in various architectures and tasks demonstrate that BackSlash can reduce memory usage by 60% - 90% without accuracy loss and provides significant compression gain compared to compression after training. Moreover, BackSlash proves to be highly versatile: it enhances generalization with small Lagrange multipliers, improves model robustness to pruning (maintaining accuracy even at 80% pruning rates), and enables network simplification for accelerated inference on edge devices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。