动态按提示剪枝，让大模型推理更省力。
$μ$-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts
- 将剪枝设计为微专家混合模型，按输入自适应选择
- 无需重训练，推理时实时降低计算量
- 适合需要快速响应的部署场景
为应对大模型带来的巨大计算负担，无需重训练的激活感知压缩技术被提出。然而，这些方法依赖校准数据，面对未知下游任务时可能因领域偏移而失效。通过高效的校准过程，激活感知剪枝可针对每个提示动态执行，从而在推理阶段实现更低的计算复杂度。本文将其建模为微专家混合（μ-MoE）。实验表明，μ-MoE能实时动态适应不同任务/提示的结构化稀疏性。
原文摘要 · Abstract (English)
To tackle the huge computational demand of large foundation models, activation-aware compression techniques without retraining have been introduced. However, since these rely on calibration data, domain shift may arise for unknown downstream tasks. With a computationally efficient calibration, activation-aware pruning can be executed for every prompt adaptively, yet achieving reduced complexity at inference. We formulate it as a mixture of micro-experts, called $μ$-MoE. Several experiments demonstrate that $μ$-MoE can dynamically adapt to task/prompt-dependent structured sparsity on the fly.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。