模型压缩到每权重1比特以下时，权值符号难以优化，导致性能瓶颈。
Sign Lock-In: Randomly Initialized Weight Signs Persist and Bottleneck Sub-Bit Model Compression
- 发现初始化符号在训练中基本不变，仅少数权重因接近零点而翻转。
- 提出符号锁死理论，解释为何低比特压缩存在无法突破的1比特墙。
- 设计新训练方法，通过低秩符号模板避免符号随机性，提升压缩效率。
子比特模型压缩旨在将每个权重的存储降至1比特以下；随着幅值被极端压缩，符号位成为固定成本瓶颈。在Transformer、CNN和MLP中，学习得到的符号矩阵难以进行低秩近似，其谱特性与独立同分布的Rademacher基线无异。这种随机性导致了子比特压缩的下限——即1比特墙。尽管看似随机，绝大多数权重仍保留初始符号；符号翻转主要发生在极少数接近零点的边界穿越事件中，表明符号模式的随机性主要源自初始化。我们提出符号锁死理论，通过随机梯度下降（SGD）噪声下的停止时间分析，证明有效符号翻转数量服从几何尾分布。基于此机制，我们引入一种从零开始的低秩符号模板训练方法，成功规避了1比特墙的出现。
原文摘要 · Abstract (English)
Sub-bit model compression targets storage below one bit per weight; as magnitudes are aggressively compressed, the sign bit becomes a fixed-cost bottleneck. Across Transformers, CNNs, and MLPs, learned sign matrices resist low-rank approximation and are spectrally indistinguishable from an i.i.d. Rademacher baseline. This randomness gives rise to the lower bound of sub-bit model compression -- the one-bit wall. Despite this apparent randomness, most weights retain their initialization signs; flips primarily occur via rare near-zero boundary crossings, suggesting that sign-pattern randomness is largely inherited from initialization. We formalize this behavior with sign lock-in theory, a stopping-time analysis of sign flips under SGD noise. Under bounded updates and a rare re-entry condition into a small neighborhood of zero, the number of effective sign flips exhibits a geometric tail. Building on this mechanism, we introduce a from-scratch low-rank sign-template training method that prevents the emergence of this one-bit wall.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。