针对领域嵌入模型的压缩难题,提出基于梯度对齐的精准剪枝方法。
GAPrune: Gradient-Alignment Pruning for Domain-Aware Embeddings
- 利用梯度对齐与费雪信息融合评估参数重要性
- 50%稀疏度下性能仅比完整模型低2.5%
- 适合资源受限环境下需强领域特性的应用
领域特定嵌入模型在代码代理、金融检索等任务中表现优异,但通常基于参数量达数十亿的大型语言模型,难以部署于资源受限环境。现有剪枝方法对所有参数一视同仁,无法区分通用语义与领域特征,导致剪枝效果不佳。为此,本文提出GAPrune,通过费雪信息衡量重要性,结合通用与领域梯度对齐评估参数行为,构建领域对齐重要性(DAI)评分。低DAI值表示该参数对领域任务不关键或与通用目标冲突。在FinMTEB和ChemTEB两个领域基准上,单次剪枝至50%稀疏度时,性能损失小于2.5%,优于所有基线。经100步微调后,在FinMTEB上提升4.51%,ChemTEB上提升1.73%,证明该方法不仅能保持,还能增强领域能力。结果表明,合理的剪枝策略可实现压缩与领域专化双重提升。
原文摘要 · Abstract (English)
Domain-specific embedding models have shown promise for applications that require specialized semantic understanding, such as coding agents and financial retrieval systems, often achieving higher performance gains than general models. However, state-of-the-art embedding models are typically based on LLMs, which contain billions of parameters, making deployment challenging in resource-constrained environments. Model compression through pruning offers a promising solution, but existing pruning methods treat all parameters uniformly, failing to distinguish between general semantic representations and domain-specific patterns, leading to suboptimal pruning decisions. Thus, we propose GAPrune, a pruning framework that addresses this challenge by considering both domain importance and preserving general linguistic foundation. Our method uses Fisher Information to measure importance and general-domain gradient alignment to assess parameter behavior, then combines these signals using our Domain Alignment Importance (DAI) scoring. Lower DAI scores indicate that the parameter is either less important for the domain task or creates conflicts between domain and general objectives. Experiments on two domain benchmarks, FinMTEB and ChemTEB, show that GAPrune maintains performance within 2.5% of dense models in one-shot pruning at 50% sparsity, while outperforming all baselines. With retraining in 100 steps, GAPrune achieves +4.51% improvement on FinMTEB and +1.73% on ChemTEB, demonstrating that our pruning strategy not only preserves but enhances domain-specific capabilities. Our findings demonstrate that principled pruning strategies can achieve model compression and enhanced domain specialization, providing the research community with a new approach for development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。