用大模型自动调参，让AI模型在不同硬件上更快更准运行。
From Bits to Chips: An LLM-based Hardware-Aware Quantization Agent for Streamlined Deployment of LLMs
- 用大模型做量化调优，自动匹配硬件最优参数。
- 在Llama上实现2.3倍推理加速，吞吐量和精度双提升。
- 适合想快速部署AI模型但不懂硬件优化的用户。
部署大型语言模型（LLMs）正吸引越来越多非专业用户，但受限于硬件资源，如何在保持高精度的同时满足计算与内存需求仍是挑战。模型量化可缓解存储与算力瓶颈，但调参与部署复杂度高，使过程对多数用户不友好。本文提出硬件感知量化代理（HAQA），一个基于大模型的自动化框架，通过高效超参数调优与硬件配置，简化整个量化与部署流程，显著提升易用性与部署质量。实验表明，相比未优化模型，HAQA在Llama上实现最高2.3倍推理速度提升，同时提高吞吐量与准确性。该框架支持跨平台自适应量化策略，能自动发现看似反直觉但最优的设置，大幅减少人工干预，展现出卓越的适应能力。代码将公开。
原文摘要 · Abstract (English)
Deploying models, especially large language models (LLMs), is becoming increasingly attractive to a broader user base, including those without specialized expertise. However, due to the resource constraints of certain hardware, maintaining high accuracy with larger model while meeting the hardware requirements remains a significant challenge. Model quantization technique helps mitigate memory and compute bottlenecks, yet the added complexities of tuning and deploying quantized models further exacerbates these challenges, making the process unfriendly to most of the users. We introduce the Hardware-Aware Quantization Agent (HAQA), an automated framework that leverages LLMs to streamline the entire quantization and deployment process by enabling efficient hyperparameter tuning and hardware configuration, thereby simultaneously improving deployment quality and ease of use for a broad range of users. Our results demonstrate up to a 2.3x speedup in inference, along with increased throughput and improved accuracy compared to unoptimized models on Llama. Additionally, HAQA is designed to implement adaptive quantization strategies across diverse hardware platforms, as it automatically finds optimal settings even when they appear counterintuitive, thereby reducing extensive manual effort and demonstrating superior adaptability. Code will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。