让大模型高效适配领域，一次剪枝即完成，性能接近完整模型。
EfficientXpert: Efficient Domain Adaptation for Large Language Models via Propagation-Aware Pruning
- 基于传播感知准则自动筛选可剪枝权重，无需反向传播。
- 在40%稀疏度下，医疗法律任务性能达密集模型的98%。
- 训练时间与LoRA相当，显存占用仅高1%，适合资源受限部署。
大语言模型（LLMs）正被广泛用于法律、医疗、金融等领域的定制化应用。然而其规模限制了在资源受限环境中的部署，现有压缩方法常在领域适配后性能下降，或需大量额外计算。我们提出EfficientXpert，一种轻量级领域剪枝框架，包含ForeSight Mask（无反向传播的传播感知剪枝准则）和Partial Brain Surgeon（固定稀疏模式下的低秩适配器闭式更新）。该方法以与标准LoRA相当的微调成本，通过单次剪枝将通用预训练模型转化为稀疏的领域专家。在医疗与法律基准测试中，EfficientXpert在40%稀疏度下达到密集模型98%的性能，优于先前剪枝基线，同时保持与LoRA相当的训练时间，并在峰值显存上仅超出1%。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly adapted into domain-specific variants for applications in law, healthcare, and finance. Their scale, however, limits deployment in resource-constrained settings, and existing compression approaches often either degrade after domain adaptation or require substantial additional computation. We introduce EfficientXpert, a lightweight framework for domain pruning that integrates ForeSight Mask, a propagation-aware criterion for selecting weights to prune without backpropagation, and Partial Brain Surgeon, an efficient closed-form update for low-rank adapters under a fixed sparsity pattern. With fine-tuning cost comparable to standard LoRA, EfficientXpert converts a general pretrained model into a sparse, domain-adapted expert in a single pruning step. Across health and legal benchmarks, EfficientXpert reaches up to 98 percent of dense performance at 40 percent sparsity, improving over prior pruning baselines while matching LoRA training time and staying within 1 percent of LoRA peak GPU memory in our experiments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。