提出自适应对数空间量化，让优化器更省内存且精度更高。
Beyond Dense Adam States: Adaptive Log-Space Quantization for Memory-Efficient Optimizers
- 基于对数空间自适应划分量化区间，精准保留非零值和零值。
- 在TinyLlama上实现72.90困惑度，优于8位AdamW的73.54。
- 适用于因子分解、置信度调节等新型优化器，适合大模型训练场景。
优化器状态量化通常针对Adam的密集、参数对齐的一阶和二阶矩数组设计。这一抽象在内存高效优化器中失效，因为其状态可能是因子分解、置信度调制或投影空间维护的，相同的重建误差可能导致不同的更新误差。我们提出将优化器状态量化建模为表示、拓扑与更新语义的联合问题。引入自适应对数空间（AL）量化方法，适用于非负状态：每个块根据观测到的非零对数区间自适应调整量化范围，并为零值单独保留码字，确保 $q = 0 \Leftrightarrow x = 0$；符号动量与状态精度可独立选择。受控实验表明，自适应范围降低更新误差与时间漂移，零值保留维持休眠状态，状态拓扑限制有效块粒度。端到端语言模型训练评估了该策略在密集、因子分解、置信度和投影优化器状态上的表现。在TinyLlama-1.1B上，使用均匀8位动量的AL8达到72.90困惑度，优于bitsandbytes 8位AdamW的73.54，且优化器状态存储相近、吞吐更高。当非负状态采用AL16时，CAME在三个随机种子下均达到参考级最终困惑度，而语义分组与保护策略弥补了量化Adafactor在10万步后的大部分损失差距。这些结果表明，状态拓扑与更新语义应作为优化器量化设计的核心约束。
原文摘要 · Abstract (English)
Optimizer-state quantization is commonly designed for Adam's dense, parameter-aligned first- and second-moment arrays. This abstraction breaks for memory-efficient optimizers, whose states may be factored, confidence-modulated, or maintained in a projected space, so similar reconstruction error can produce different update error. We formulate optimizer-state quantization as a joint problem over representation, topology, and update semantics. We then introduce Adaptive Log-Space (AL) quantization for non-negative states. AL fits each block's observed nonzero logarithmic interval and reserves a separate code for exact zero, enforcing $q = 0 \Leftrightarrow x = 0$; signed momentum and state precision remain independently selectable. Controlled probes show that adaptive ranges reduce update error and temporal drift, exact-zero reservation preserves dormant states, and state topology constrains useful block granularity. End-to-end language-model training evaluates the resulting policy across dense, factored, confidence, and projected optimizer states. On TinyLlama-1.1B, AL8 with uniform 8-bit momentum reaches 72.90 perplexity versus 73.54 for bitsandbytes 8-bit AdamW, with comparable optimizer-state storage and higher throughput. CAME matches reference-level final perplexity across three seeds when its non-negative states use AL16, while a semantic grouping-and-protection policy closes most of quantized Adafactor's 100K-step late-loss gap. These results make state topology and update semantics first-class design constraints for optimizer quantization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。