用小数据找量化策略，高效适配大模型。
Learning from Loss Landscape: Generalizable Mixed-Precision Quantization via Adaptive Sharpness-Aware Gradient Aligning
- 在小数据上搜索量化方案，再迁移至大规模数据
- 在ImageNet上达到同等精度，计算成本大幅降低
- 自适应扰动半径加速优化，适合资源受限场景
混合精度量化（MPQ）通过为每层确定最优位宽，成为优化神经网络的关键技术。现有方法需在大规模数据集上进行昂贵的量化策略搜索。本文提出新方法：先在小数据集上搜索量化策略，再泛化到大规模数据集，简化流程，无需大规模量化微调，仅需调整模型权重。核心技术包括：基于尖锐度感知的最小化以增强泛化能力，隐式梯度方向对齐解决多目标优化中的梯度冲突，以及自适应扰动半径加速优化。理论分析与实验均验证有效性。仅使用CIFAR10（仅为ImageNet训练数据的0.5%）进行策略搜索，在ImageNet上实现等效精度，计算成本显著降低，效率最高提升150%超过基线。
原文摘要 · Abstract (English)
Mixed Precision Quantization (MPQ) has become an essential technique for optimizing neural network by determining the optimal bitwidth per layer. Existing MPQ methods, however, face a major hurdle: they require a computationally expensive search for quantization policies on large-scale datasets. To resolve this issue, we introduce a novel approach that first searches for quantization policies on small datasets and then generalizes them to large-scale datasets. This approach simplifies the process, eliminating the need for large-scale quantization fine-tuning and only necessitating model weight adjustment. Our method is characterized by three key techniques: sharpness-aware minimization for enhanced quantization generalization, implicit gradient direction alignment to handle gradient conflicts among different optimization objectives, and an adaptive perturbation radius to accelerate optimization. Both theoretical analysis and experimental results validate our approach. Using the CIFAR10 dataset (just 0.5\% the size of ImageNet training data) for MPQ policy search, we achieved equivalent accuracy on ImageNet with a significantly lower computational cost, while improving efficiency by up to 150% over the baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。