TwIST通过并行训练子网络，实现零成本剪枝，大幅降低大模型推理开销。
TwIST: Rigging the Lottery in Transformers with Independent Subnetwork Training
- 并行训练多个子网络，周期性聚合参数并重采样，自动发现高质量稀疏结构。
- 在50%以上稀疏度下，困惑度达23.14，优于此前最优方法（31.64）。
- 生成结构化稠密矩阵，适合普通硬件加速，无需额外微调或恢复步骤。
我们提出TwIST，一种用于高效大语言模型稀疏化的分布式训练框架。该框架并行训练多个子网络，周期性聚合参数并重采样新子网络。该过程无需后训练校准或基于海森矩阵的恢复，即可识别高质量子网络（“黄金票”）。因此，TwIST在部署时实现零成本剪枝，同时达到与当前最先进后训练稀疏化方法相当的困惑度表现。在高稀疏度（如50%以上）下优势显著，例如在50%稀疏度下达到23.14困惑度，优于最近方法的31.64。与非结构化剪枝不同，TwIST生成结构化稠密矩阵，可在不支持高效稀疏计算的通用硬件（如CPU）上实现实际推理加速和内存压缩。该方法为部署可直接使用的稀疏大模型提供了高效的训练路径，无需额外微调或恢复开销。
原文摘要 · Abstract (English)
We introduce TwIST, a distributed training framework for efficient large language model (LLM) sparsification. TwIST trains multiple subnetworks in parallel, periodically aggregates their parameters, and resamples new subnetworks during training. This process identifies high-quality subnetworks ("golden tickets") without requiring post-training procedures such as calibration or Hessian-based recovery. As a result, TwIST enables zero-cost pruning at deployment time while achieving perplexity competitive with state-of-the-art post-training sparsification methods. The benefits are most pronounced under aggressive sparsity (e.g., 50%+), where TwIST significantly outperforms baseline methods; for example, reaching 23.14 PPL compared to 31.64 for the closest prior approach. Unlike unstructured pruning, TwIST produces structured, dense matrices that offer practical inference speedups and memory reductions on commodity hardware (e.g., CPUs) that do not support efficient sparse computation. TwIST provides an efficient training-time path to deployable sparse LLMs without additional fine-tuning or recovery overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。