用优化蒸馏与量化压缩大模型，让小设备也能高效运行。
Advancing Model Refinement: Muon-Optimized Distillation and Quantization for LLM Deployment
- 结合数据蒸馏、低秩适配和量化技术，减少模型体积。
- 实现最高2倍内存压缩（6GB→3GB），推理速度更快。
- 适合资源受限场景，如移动端或边缘设备部署。
大型语言模型（LLMs）虽能实现先进自然语言处理，但在资源受限的边缘设备上部署时面临计算、内存和能耗过高难题。解决这一问题需应对三个关键挑战：获取任务特定数据、微调以提升性能、压缩模型以加速推理并降低资源消耗。本文提出一种集成框架，融合基于GPTQ的量化、低秩适配（LoRA）以及专有的数据蒸馏流程，在显著降低模型大小与复杂度的同时，保持甚至提升任务特定性能。通过数据蒸馏、基于KL散度的知识蒸馏、贝叶斯超参数优化及Muon优化器，该流程实现最高2倍内存压缩（例如将6GB模型压缩至3GB），并支持高效专用任务推理。实验结果表明，其在标准LLM基准测试中表现优于仅使用GPTQ量化的方法，其中Muon优化器显著提升了微调模型在量化过程中的抗精度衰减能力。
原文摘要 · Abstract (English)
Large Language Models (LLMs) enable advanced natural language processing but face deployment challenges on resource-constrained edge devices due to high computational, memory, and energy demands. Optimizing these models requires addressing three key challenges: acquiring task-specific data, fine-tuning for performance, and compressing models to accelerate inference while reducing resource demands. We propose an integrated framework combining GPTQ-based quantization, low-rank adaptation (LoRA), and a specialized data distillation process to significantly reduce model size and complexity while preserving or enhancing task-specific performance. By leveraging data distillation, knowledge distillation via Kullback-Leibler divergence, Bayesian hyperparameter optimization, and the Muon optimizer, our pipeline achieves up to 2x memory compression (e.g., reducing a 6GB model to 3GB) and enables efficient inference for specialized tasks. Empirical results demonstrate superior performance on standard LLM benchmarks compared to GPTQ quantization alone, with the Muon optimizer notably enhancing fine-tuned models' resistance to accuracy decay during quantization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。