量化会降低Transformer表达能力，一比特损失可能让模型无法完成精确匹配任务。
Every Bit Counts: A Theoretical Study of Precision-Expressivity Tradeoffs in Quantized Transformers
- 通过构造特定函数，证明一比特精度下模型无法完成某些比较任务。
- 实验显示仅降一比特就可能导致模型完全失去表达关键逻辑的能力。
- 对需要精确匹配的任务（如查重、归属判断）有重要指导意义。
量化通过降低Transformer计算的数值精度来加速推理，但其对模型表达能力的影响尚不明确。本文揭示了表达能力与精度间的细粒度理论权衡：对任意精度p，我们构造了一个受等值函数启发的函数Γ，证明单层Softmax Transformer在使用p比特精度时可计算Γ，但在仅用p-1比特时则无法计算。这一结果直观解释了量化导致表达能力下降的常见现象。实际应用中表明，需要等值比较的任务（如精确匹配、成员判断等）对量化特别敏感，丢失一比特就可能跨越模型无法可靠表示该比较的临界点。因此，该研究为实践者提供了选择量化程度的依据：精度应根据任务中需检查的等值长度动态设定。证明结合了显式有限精度Transformer构造与通信复杂性下界分析，得到紧致的‘一比特’阈值。
原文摘要 · Abstract (English)
Quantization reduces the numerical precision of Transformer computations and is widely used to accelerate inference, yet its effect on expressivity remains poorly characterized. We demonstrate a fine-grained theoretical tradeoff between expressivity and precision: For every p we exhibit a function Γ, inspired by the equality function, and prove that a one-layer softmax Transformer can compute Γ, with p bits of precision, but not with p-1 bits of precision. This result concretely explains the widely observed phenomenon of empirical loss of expressivity when quantization is used. Practically, it suggests that tasks requiring equality-like comparisons (exact match, membership, etc.) are especially sensitive to quantization. Dropping even one bit can cross a threshold where the model cannot represent the needed comparison reliably. Thus, it paves the way for developing heuristics that will help practitioners choose how much quantization is possible: the precision should be chosen as a function of the length of equality to be checked for the specific task. Our proofs combine explicit finite-precision Transformer constructions with communication-complexity lower bounds, yielding a tight "one-bit" threshold.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。