arXiv:2604.13440cs.LGcs.AI2026-04被引 1

用KL散度快速评估混合模型各部分对量化敏感度,实现边缘设备高效部署。

A KL Lens on Quantization: Fast, Forward-Only Sensitivity for Mixed-Precision SSM-Transformer Models

论文配图:A KL Lens on Quantization: Fast, Forward-Only Sensitivity for Mixed-Precision SSM-Transformer Models
图 1 · 摘自论文原文
  • 基于前向传播的KL散度分析,无需反向传播和重训练
  • 在混合SSM-Transformer模型上实现接近FP16精度的性能,仅需INT4量化
  • 适用于隐私受限或无领域数据场景,适合资源受限边缘设备部署

将大语言模型(LLMs)部署于边缘设备面临严重的计算与内存限制,制约实时处理与本地智能。结合结构化状态空间模型(SSMs)与Transformer架构的混合模型在效率与性能间取得平衡。激进量化可显著压缩模型体积并加速推理,但不同组件对量化敏感度不均,需精细管理。本文提出一种轻量级、无需反向传播、基于代理的敏感性分析框架,用于识别混合SSM-Transformer组件中对量化最敏感的部分。该方法仅依赖前向传播指标,避免高成本梯度计算与再训练,适用于因隐私或版权限制而无法获取领域数据的情况。我们还提供了形式化分析,表明对于语言建模任务,Kullback-Leibler(KL)散度比广泛使用的均方误差(MSE)和信噪比(SQNR)更优地捕捉量化敏感性。在SSM及混合架构上的大量实验表明,基于KL的排序与实际性能下降高度一致,优于其他指标。该框架使先进混合模型能在资源受限边缘设备上以最小精度损失实现实用部署。我们在Intel Lunar Lake硬件上进行真实设备性能测试,验证了KL指导的混合精度量化在CPU与GPU模式下均能实现接近FP16的困惑度,且模型尺寸与吞吐量媲美统一INT4。代码已开源:https://github.com/jasonkongie/kl-ssm-quant。

原文摘要 · Abstract (English)

Deploying Large Language Models (LLMs) on edge devices faces severe computational and memory constraints, limiting real-time processing and on-device intelligence. Hybrid architectures combining Structured State Space Models (SSMs) with transformer-based LLMs offer a balance of efficiency and performance. Aggressive quantization can drastically cut model size and speed up inference, but its uneven effects on different components require careful management. In this work, we propose a lightweight, backpropagation-free, surrogate-based sensitivity analysis framework to identify hybrid SSM-Transformer components most susceptible to quantization-induced degradation. Relying solely on forward-pass metrics, our method avoids expensive gradient computations and retraining, making it suitable for situations where access to in-domain data is limited due to proprietary restrictions or privacy constraints. We also provide a formal analysis showing that the Kullback-Leibler (KL) divergence metric better captures quantization sensitivity for Language modeling tasks than widely adopted alternatives such as mean squared error (MSE) and signal-to-quantization-noise ratio (SQNR). Through extensive experiments on SSM and hybrid architectures, our ablation studies confirm that KL-based rankings align with observed performance drops and outperform alternative metrics. This framework enables the practical deployment of advanced hybrid models on resource-constrained edge devices with minimal accuracy loss. We further validate our approach with real-world on-device profiling on Intel Lunar Lake hardware, demonstrating that KL-guided mixed-precision achieves near-FP16 perplexity with model sizes and throughput competitive with Uniform INT4 on both CPU and GPU execution modes. Code is available at https://github.com/jasonkongie/kl-ssm-quant.

量化边缘计算混合模型KL散度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。