针对智能体大模型推理瓶颈,分阶段量化加速,预填充提速3倍且不损失精度。
Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs

- 分阶段量化:预填充用高吞吐FP4,解码保留BF16精度
- 实测预填充阶段速度提升最高3倍,任务性能基本不变
- 适合长上下文、多轮交互的智能体应用,部署友好
大模型智能体通过规划、工具调用和多步交互解决复杂任务,但其工作流常带来显著输入端开销,使计算密集的预填充阶段成为长上下文、多轮推理的主要瓶颈。本文提出Mix-Quant,一种简单有效的分阶段感知量化框架,用于加速智能体推理。我们研究了在智能体工作流中使用FP4量化,发现对整个推理过程进行量化会带来显著性能下降。相反,预填充阶段存在大量量化冗余,即使在高量化强度下也仅造成极小精度损失,尽管它是主要计算来源。基于此,我们在预填充阶段采用高吞吐的NVFP4量化,而解码阶段保持BF16精度。通过将预填充加速与解码质量解耦,Mix-Quant结合了分阶段算法量化与硬件高效的NVFP4执行,有效缓解了大模型智能体的推理瓶颈。在长上下文和智能体基准上的大量实验表明,Mix-Quant在基本保持任务性能的同时,实现了显著的效率提升,预填充阶段最高可达3倍加速。
原文摘要 · Abstract (English)
LLM agents have recently emerged as a powerful paradigm for solving complex tasks through planning, tool use, memory retrieval, and multi-step interaction. However, these agentic workflows often introduce substantial input-side overhead, making the compute-intensive prefilling stage a key bottleneck in long-context, multi-turn inference. In this work, we propose Mix-Quant, a simple and effective phase-aware quantization framework for fast agentic inference. We first investigate FP4 quantization in agentic LLM workflows and observe that quantizing the entire inference process can incur significant performance degradation. In contrast, the prefilling stage exhibits substantial quantization redundancy and can therefore be quantized with minimal accuracy loss, despite being the dominant source of computation. Based on this insight, we apply high-throughput NVFP4 quantization to the prefilling phase while preserving BF16 precision for decoding. By decoupling prefilling acceleration from decoding quality, Mix-Quant combines phase-aware algorithmic quantization with hardware-efficient NVFP4 execution to alleviate the inference bottleneck in LLM agents. Extensive experiments across long-context and agentic benchmarks demonstrate that Mix-Quant largely preserves task performance while delivering significant efficiency improvements, achieving up to a 3x speedup during prefilling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。