用π的无理数特性实现3.32位参数压缩，内存减40%仍保精度
Irrational Complex Rotations Empower Low-bit Optimizers
- 将参数映射到复数空间，用旋转角量化表示
- 实现3.32位参数压缩，显存减少40%且精度不变
- 适合大模型训练中内存受限场景
本文提出一种新型优化器状态压缩算法π-Quant，利用无理数（如π）的性质实现高效内存训练。核心思想基于数学发现：一对参数可通过复数旋转方案用单一旋转角表示。据此，我们将参数映射至复数空间，并以对应旋转角进行量化。为高效集成至优化过程，我们设计了一套线性复杂度的几何方程系统，精确计算旋转角。在多种任务上的实验表明，π-Quant可将参数位宽降至3.32位，参数规模缩小75%，GPU显存使用降低40%，同时保持完整精度。
原文摘要 · Abstract (English)
In this paper, we propose a novel optimizer state compression algorithm, namely $π$-Quant, which leverages the properties of irrational numbers (e.g., $π$) for memory-efficient training. The core idea is based on our mathematical findings, which show that a pair of parameters can be represented by a single rotation angle using the complex rotation scheme. Building on this insight, we map the parameters into a complex space and perform quantization using the corresponding rotation angles. To efficiently integrate it into optimization process, we develop an efficient system of geometric equations that computes the precise rotation angles with linear complexity. We evaluate $π$-Quant on a wide range of tasks. Our experiments show that it can reduce the bit-width of parameters to 3.32-bit, achieving a 75% reduction in parameter scale and a 40% decrease in GPU memory usage, all while maintaining full accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。