联邦剪枝让大模型压缩更私密，无需共享原始数据
Exploring Federated Pruning for Large Language Models
- 各客户端仅需本地计算剪枝掩码上传，保护数据隐私
- 单次剪枝+层间比较+不缩放权重效果最佳
- 适合医疗、金融等隐私敏感领域的模型压缩
大模型剪枝是压缩大语言模型以实现在资源受限设备上部署的有前景技术。然而，现有方法通常需要访问公开校准数据，这在隐私敏感领域难以实现。为此，我们提出FedPrLLM，一种全面的联邦剪枝框架，用于在保护隐私的前提下压缩大语言模型。在该框架中，每个客户端仅需基于本地校准数据计算剪枝掩码矩阵，并将其共享给服务器以剪枝全局模型。此方法在保留各客户端知识的同时，确保本地数据隐私。我们还进行了广泛实验，探索了不同对比组、剪枝策略及权重缩放决策。评估结果表明，在FedPrLLM框架中，单次剪枝结合层间比较且不进行权重缩放是最优选择。希望本工作能为隐私敏感领域的大模型剪枝研究提供指导。代码已开源：https://github.com/Pengxin-Guo/FedPrLLM。
原文摘要 · Abstract (English)
LLM pruning has emerged as a promising technology for compressing LLMs, enabling their deployment on resource-limited devices. However, current methodologies typically require access to public calibration samples, which can be challenging to obtain in privacy-sensitive domains. To address this issue, we introduce FedPrLLM, a comprehensive federated pruning framework designed for the privacy-preserving compression of LLMs. In FedPrLLM, each client only needs to calculate a pruning mask matrix based on its local calibration data and share it with the server to prune the global model. This approach allows for collaborative pruning of the global model with the knowledge of each client while maintaining local data privacy. Additionally, we conduct extensive experiments to explore various possibilities within the FedPrLLM framework, including different comparison groups, pruning strategies, and the decision to scale weights. Our extensive evaluation reveals that one-shot pruning with layer comparison and no weight scaling is the optimal choice within the FedPrLLM framework. We hope our work will help guide future efforts in pruning LLMs in privacy-sensitive fields. Our code is available at https://github.com/Pengxin-Guo/FedPrLLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。