AutoMixQ动态优化量化配置，让大模型微调更省内存、性能更强
AutoMixQ: Self-Adjusting Quantization for High Performance Memory-Efficient Fine-Tuning
- 按层自适应选择量化方案，避免统一量化导致的性能下降
- 在LLaMA-7B上30%剪枝时准确率达66.21%，内存降低超27%
- 适合资源受限下追求高性能微调的研究者和工程师
在资源受限条件下微调大语言模型是深度学习的重要挑战。低秩适配（LoRA）、剪枝和量化虽能提升资源效率，但直接组合常导致性能不佳，尤其在各层统一量化时更为明显。这源于剪枝引入的复杂层间关系，需更精细的量化策略。为此，我们提出AutoMixQ，一个端到端优化框架，为每层模型自动选择最优量化配置。该方法利用轻量级性能模型引导选择过程，显著减少搜索时间和计算开销。通过引入帕累托最优，AutoMixQ在内存使用与性能间取得平衡，在严格资源约束下逼近模型能力上限。实验表明，其在多个基准测试中均实现更低内存消耗与更高性能。例如，在LLaMA-7B上30%剪枝时，BoolQ准确率达66.21%，优于LoRA的62.45%和LoftQ的58.96%，内存比LoRA减少35.5%，比LoftQ减少27.5%。
原文摘要 · Abstract (English)
Fine-tuning large language models (LLMs) under resource constraints is a significant challenge in deep learning. Low-Rank Adaptation (LoRA), pruning, and quantization are all effective methods for improving resource efficiency. However, combining them directly often results in suboptimal performance, especially with uniform quantization across all model layers. This is due to the complex, uneven interlayer relationships introduced by pruning, necessitating more refined quantization strategies. To address this, we propose AutoMixQ, an end-to-end optimization framework that selects optimal quantization configurations for each LLM layer. AutoMixQ leverages lightweight performance models to guide the selection process, significantly reducing time and computational resources compared to exhaustive search methods. By incorporating Pareto optimality, AutoMixQ balances memory usage and performance, approaching the upper bounds of model capability under strict resource constraints. Our experiments on widely used benchmarks show that AutoMixQ reduces memory consumption while achieving superior performance. For example, at a 30\% pruning rate in LLaMA-7B, AutoMixQ achieved 66.21\% on BoolQ compared to 62.45\% for LoRA and 58.96\% for LoftQ, while reducing memory consumption by 35.5\% compared to LoRA and 27.5\% compared to LoftQ.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。