arXiv:2508.03332cs.LGcs.AI2025-08ACL被引 3

按层分配比特数,让小模型在极低精度下仍保持高精度。

Exploring Layer-wise Information Effectiveness for Post-Training Quantization in Small Language Models

  • 按层重要性动态分配比特,不改计算核,适合边缘设备。
  • 2比特压缩下,比传统方法在多个小模型上提升准确率。
  • 无需反向传播或复杂测试,自动分配比特,省时省力。

参数量小于80亿的大语言模型常存在过度冗余:许多层贡献信息有限,却主导推理时的内存与能耗。本文提出LieQ——一种面向硬件、基于度量的后训练量化框架,解决小模型在极端低比特压缩下的精度保持难题。该方法在每层内保持统一比特数,跨层混合精度,保留标准乘法核,避免不规则内存访问、查表或格式。研究发现,层级功能显著性与表征紧凑性强相关,训练能量集中度高的层不可替代。基于此,提出纯几何驱动的敏感度代理,无需梯度更新或困惑度探测,即可在目标平均比特预算下自动分配比特。在低于2比特时,LieQ在Qwen3和LLaMA3.x系列模型上持续缩小传统2比特基线的精度差距,同时保持标准核效率。该方法为资源受限边缘设备部署小模型提供了实用路径。代码将公开于 https://github.com/HeXiao-55/LieQ-official.git。

原文摘要 · Abstract (English)

Large language models with billions of parameters are often over-provisioned: many layers contribute little unique information yet dominate the memory and energy footprint during inference. We present LieQ Layer-wise information effectiveness Quantization, a hardware-native, metric-driven post-training quantization framework that addresses the critical challenge of maintaining accuracy in sub-8B models, model parameters less than 8B, under extreme low-bit compression. LieQ keeps uniform bit-width within each layer while mixing precision across layers, preserving standard multiplication kernels and avoiding irregular memory access, codebooks, or irregular formats at inference time. Our method uncovers a strong correlation between layer-wise functional saliency and representational compactness, revealing that layers with higher training-induced energy concentration are functionally irreplaceable. Leveraging this insight, we propose a purely geometry-driven sensitivity proxy that enables automatic bit-width allocation under a target average-bit budget without expensive gradient updates or inference-based perplexity probing. At sub 2-bit, LieQ consistently reduces the large accuracy gap typically observed for naive 2-bit baselines on Qwen3 and LLaMA3.x families, while retaining standard-kernel efficiency. These properties make LieQ a practical path toward deploying small language models on resource-constrained edge devices. Code will available here: https://github.com/HeXiao-55/LieQ-official.git.

量化小模型边缘计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。