用低秩适配器+神经架构搜索，让大模型更轻更快
Low-Rank Adapters Meet Neural Architecture Search for LLM Compression
- 将低秩适配器与神经架构搜索结合，实现高效参数压缩
- 在保持性能的同时，内存占用减少、推理速度提升
- 适合边缘设备部署，推动大模型落地实用场景
大语言模型（LLMs）的快速扩张带来了微调和部署所需的大量计算资源挑战。近期低秩适配器在参数高效微调（PEFT）方面表现出色。本文系统探讨了将低秩表示与神经架构搜索（NAS）技术，特别是权重共享超网络相结合的创新方法。通过整合这些策略，开发出鲁棒的大型预训练模型压缩与微调解决方案。分析表明，这些组合方法具有潜力推动大模型普及，使其在资源受限环境下更易部署。所得模型具备更小的内存开销和更快的推理速度，为大模型的实际化、可扩展应用铺平道路。代码与模型已开源：https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning。
原文摘要 · Abstract (English)
The rapid expansion of Large Language Models (LLMs) has posed significant challenges regarding the computational resources required for fine-tuning and deployment. Recent advancements in low-rank adapters have demonstrated their efficacy in parameter-efficient fine-tuning (PEFT) of these models. This retrospective paper comprehensively discusses innovative approaches that synergize low-rank representations with Neural Architecture Search (NAS) techniques, particularly weight-sharing super-networks. Robust solutions for compressing and fine-tuning large pre-trained models are developed by integrating these methodologies. Our analysis highlights the potential of these combined strategies to democratize the use of LLMs, making them more accessible for deployment in resource-constrained environments. The resulting models exhibit reduced memory footprints and faster inference times, paving the way for more practical and scalable applications of LLMs. Models and code are available at https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。