arXiv:2608.26581cs.LG2026-08

4-bit量化会严重损害多模态大模型性能,提出轻量级修复方案恢复精度。

Activation Outliers Matter: Robust Recovery for Quantized Multimodal LLMs

论文配图:Activation Outliers Matter: Robust Recovery for Quantized Multimodal LLMs
图 1 · 摘自论文原文
  • 用辅助残差路径重建量化激活值,提升低比特下的表示精度
  • 在Wan2.2和Qwen3-VL上使MXFP4/HiF4的性能接近BF16基线
  • 无需修改结构,计算开销极小,适合实际部署

低比特量化为降低多模态大语言模型(MLLMs)的计算与内存开销提供了可行路径。近年来,从MXFP8到超低比特格式如MXFP4和HiF4的硬件支持加速了高效MLLM训练与部署的研究。本文系统研究了代表性MLLM在视频生成与推理任务中采用这些量化方案的表现。分析表明,MXFP8可实现近无损性能,而激进的4比特量化导致显著性能下降。通过大量消融实验,我们发现激活量化是性能损失的主要来源,其影响远大于权重量化。为此,我们提出轻量级激活重建框架Residual Fallback Quantization(RFQ),在主超低比特激活表示外增加一个辅助量化残差路径,显式建模并补偿量化误差。该方法不需架构改动,计算开销可忽略。在Wan2.2和Qwen3-VL上的大量实验表明,RFQ能持续恢复因MXFP4和HiF4量化丢失的性能,显著缩小与BF16基线在生成与推理基准间的差距。研究结果确立激活量化为超低比特MLLM的核心瓶颈,并证明基于残差的激活重建是一种有效且实用的4比特部署策略。

原文摘要 · Abstract (English)

Low-bit quantization offers a promising avenue for reducing the computational and memory demands of Multimodal Large Language Models (MLLMs). Recent hardware support for low-precision formats, ranging from MXFP8 to ultra-low-bit formats such as MXFP4 and HiF4, has accelerated research into efficient MLLM training and deployment. In this work, we present a systematic study of these quantization schemes in representative MLLMs that span both video generation and reasoning tasks. Our analysis shows that MXFP8 achieves near-lossless performance, whereas aggressive 4-bit quantization leads to significant degradation. Through extensive ablations, we identify activation quantization as the primary source of this performance loss, contributing substantially more than weight quantization. Motivated by this observation, we propose Residual Fallback Quantization (RFQ), a lightweight activation reconstruction framework that supplements the primary ulta-low-bit activation representation with an auxiliary quantized residual pathway. By explicitly modeling and compensating for quantization errors, RFQ improves activation fidelity while preserving the efficiency advantages of ultra-low-bit computation. RFQ requires no architectural modifications and incurs negligible computational overhead. Extensive experiments on Wan2.2 and Qwen3-VL demonstrate that RFQ consistently recovers a substantial portion of the performance lost under the quantization of MXFP4 and HiF4, significantly narrowing the gap to BF16 baselines across both generation and 4 reasoning benchmarks. Our findings establish activation quantization as the dominant bottleneck in ultra-low-bit MLLMs and highlight residual-based activation reconstruction as an effective and practical strategy for robust 4-bit deployment.

多模态大模型4比特量化激活重建高效部署

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。