通过联合稀疏化耦合权重矩阵，提升Transformer训练效率与模型性能。
EcoSpa: Efficient Transformer Training with Coupled Sparsity
- 联合评估并稀疏化交互的权重矩阵对，保持注意力层结构关系。
- 在LLaMA-1B上实现50%内存减少和21%加速，GPT-2-Medium压缩2.2倍且困惑度更低。
- 无需定制硬件，基于标准PyTorch，适合普通设备部署大规模模型训练。
Transformer已成为现代AI的核心架构，但其高计算需求带来严峻系统挑战。尽管稀疏训练可提升效率，现有方法未能保留注意力和前馈层中乘法交互权重矩阵间的关键结构关系，导致高稀疏度下性能下降。我们提出EcoSpa，一种高效的结构化稀疏训练方法，通过同步评估与稀疏化耦合权重矩阵对，实现行/列对齐移除，保留其交互模式。EcoSpa引入新的结构性组件重要性校准粒度，在预训练与微调场景中均支持联合估计与稀疏化。实验表明：EcoSpa使LLaMA-1B实现50%内存减少、21%训练加速；在GPT-2-Medium上实现2.2倍模型压缩，困惑度降低2.4；推理速度提升1.6倍。该方法仅使用标准PyTorch操作，无需专用硬件或内核，可在通用硬件上实现高效Transformer训练。
原文摘要 · Abstract (English)
Transformers have become the backbone of modern AI, yet their high computational demands pose critical system challenges. While sparse training offers efficiency gains, existing methods fail to preserve critical structural relationships between weight matrices that interact multiplicatively in attention and feed-forward layers. This oversight leads to performance degradation at high sparsity levels. We introduce EcoSpa, an efficient structured sparse training method that jointly evaluates and sparsifies coupled weight matrix pairs, preserving their interaction patterns through aligned row/column removal. EcoSpa introduces a new granularity for calibrating structural component importance and performs coupled estimation and sparsification across both pre-training and fine-tuning scenarios. Evaluations demonstrate substantial improvements: EcoSpa enables efficient training of LLaMA-1B with 50\% memory reduction and 21\% faster training, achieves $2.2\times$ model compression on GPT-2-Medium with $2.4$ lower perplexity, and delivers $1.6\times$ inference speedup. The approach uses standard PyTorch operations, requiring no custom hardware or kernels, making efficient transformer training accessible on commodity hardware.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。