通过结构正则化缓解视觉Transformer中的激活异常值，提升低比特量化性能。
Colinearity Decay: Training Quantization-Friendly ViTs with Outlier Decay
- 引入共线性衰减机制，抑制跨矩阵有害对齐。
- 在多个任务上实现量化精度显著提升，全精度性能不降反升。
- 无需修改架构或损失函数，训练开销极小，适合部署优化。
低比特量化是高效部署视觉Transformer的实用路径，但激活异常值增加了完全量化部署的难度。现有方法要么在后训练阶段处理量化，要么在训练中抑制大激活；然而，在视觉模型中过度限制异常值可能导致全精度与量化精度之间的权衡变差。我们提出，与其简单压制异常值，不如控制使其有害的结构放大效应。为此，我们引入共线性衰减（Colinearity-Decay, CD），一种作用于Transformer块内有序矩阵对的结构正则化器。CD惩罚有害的跨矩阵对齐，减轻极端激活，且不改变网络架构或任务损失。作为解耦更新方式，CD非侵入性强，训练开销极小。在ImageNet-1K预训练、COCO检测及下游微调中，CD一致提升了多种流水线下的量化精度，同时保持甚至改善了全精度性能。结果表明，结构正则化能有效为低比特部署准备视觉Transformer，且推理时无额外开销。
原文摘要 · Abstract (English)
Low-bit quantization is a practical route for efficiently deploying vision Transformers, yet activation outliers complicate fully quantized deployment. Existing methods either handle quantization post-training or suppress large activations during training; however, aggressively restricting outliers in vision models can lead to a poorer trade-off between full-precision and quantized accuracy. We argue that rather than simply suppressing outliers, the training objective should control the structural amplification that makes them harmful. To this end, we introduce Colinearity-Decay (CD), a structural regularizer for ordered matrix pairs within Transformer blocks. CD penalizes detrimental cross-matrix alignment and mitigates extreme activations without altering the architecture or task loss. Applied as a decoupled update, CD is non-invasive and introduces minimal training overhead. Across ImageNet-1K pre-training, COCO detection, and downstream fine-tuning, CD consistently boosts quantized accuracy across multiple pipelines while preserving, or even improving, full-precision performance. Ultimately, our results demonstrate that structural regularization effectively prepares vision Transformers for low-bit deployment with zero inference-time overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。