不重训练让大模型自适应稀疏，推理效率翻倍且性能损失小。
Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs
- 按层间特征相似度动态分配保留比例，实现轻量级稀疏化
- 在GPT-NeoX和Qwen2.5上降低超60%计算量，精度损失<3%
- 无需微调，通用性强，适合部署资源受限场景
大语言模型（LLMs）虽具备强大生成与推理能力，但其Transformer架构带来高推理开销。现有加速方法多依赖任务特定微调或从头训练，增加适配成本且跨任务适用性差。本文提出自适应深度稀疏框架（AdaDSF），将现成预训练LLM转化为深度稀疏模型，无需全量重训练。核心思想是：各层对表征变换的贡献不均，由层输入与输出隐藏状态间的余弦相似度刻画。基于此，AdaDSF通过相似度统计分配层内令牌保留率，使用轻量路由器在每层筛选有效令牌，并引入特征保持对齐目标，使稀疏模型与密集模型的中间及最终表示一致。在GPT-NeoX与Qwen2.5上的语言建模与常识推理任务中，AdaDSF显著降低推理浮点运算量（FLOPs），同时性能接近密集模型。在相同稀疏度下，相比MoD、D-LLM与DLO等强基线，其准确率下降更小。
原文摘要 · Abstract (English)
Large language models (LLMs) achieve strong generation and reasoning performance, but the Transformer architecture incurs high inference cost. Existing acceleration methods often rely on task-specific fine-tuning or training from scratch, increasing adaptation cost and limiting cross-task usability. We present an Adaptive Depth Sparse Framework (AdaDSF) that converts off-the-shelf pre-trained LLMs into depth-sparse models without full retraining. Our key insight is that layers contribute unequally to representation transformation, characterized by the cosine similarity between layer input and output hidden states. Based on this, AdaDSF assigns layer-wise token retention ratios from similarity statistics, uses a lightweight router to select informative tokens at each layer, and introduces a feature-preserving alignment objective to match intermediate and final representations between sparse and dense models. On GPT-NeoX and Qwen2.5 over language modeling and commonsense reasoning, AdaDSF substantially reduces inference FLOPs while preserving performance close to dense counterparts. Under comparable sparsity, AdaDSF consistently yields smaller accuracy degradation than strong baselines including MoD, D-LLM, and DLO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。