揭示低比特优化的理论极限，给出通信与统计下界。
Information-Theoretic Lower Bounds for Bit-Constrained Stochastic Optimization via a Reduction to Compressed Gaussian Mean Estimation
- 将低比特优化转化为压缩高斯均值估计问题
- 得出通信量与迭代次数的紧致下界，依赖维度和噪声
- 适用于分析低精度训练的理论性能边界
低精度预训练（如FP8、MXFP4、NVFP4）已成为前沿语言模型的标准，但现有研究多聚焦于可实现性（算法与经验缩放律），缺乏对信息论上可能性的刻画。本文研究一个B比特量化随机一阶查询机：优化器进行T轮交互，每轮接收一个由公共随机源生成的B比特自适应梯度描述。核心贡献是将强凸二次函数优化精确还原为交互式压缩高斯均值估计问题——在该B比特查询下，查询不携带有效信息，优化等价于序列分布式估计。由此导出两个无条件下界：通信下界TB = Ω(d)，统计下界T = Ω(σ²d / ε²)，以及紧致的乘积形式下界T = Ω((σ²d / ε²) max{1, d/B})。该乘积形式亦无条件成立：一个B比特会话最多携带O(TB / σ²)的费舍尔迹信息，因此比特数而非维度限制可恢复信息；结合多元van Trees不等式，直接获得下界，无需有界似然比截断。本文还给出了近似匹配的可实现性结果，带精确逐轮比特计数，在有界动态范围查询机下，误差仅差一个对数因子；而下界针对真正的高斯（无界）梯度，此查询差距尚待解决。序列率失真视角扩展了还原至相关及漂移查询机的情形，并修正了早期猜想：正噪声相关使下界上升为(1+ρ)/(1−ρ)，而非降低。这些下界为任意低比特梯度路径提供了信息论基准，而非对当前部署的FP4系统做出最优性声明。
原文摘要 · Abstract (English)
Low-precision pretraining (FP8, MXFP4, NVFP4) is now standard for frontier language models, yet the literature is almost entirely achievability -- algorithms and empirical scaling laws -- with no matching characterization of what is information-theoretically possible. We study a B-bit quantized stochastic first-order oracle: an optimizer interacts for T rounds and receives, each round, a B-bit adaptive public-coin description of its stochastic gradient. Our main contribution is an exact reduction from optimizing a strongly convex quadratic family to interactively compressed Gaussian mean estimation -- under the B-bit oracle the query carries no information, so optimization collapses exactly onto a sequential distributed-estimation problem. This yields two unconditional lower bounds, a communication bound TB = Omega(d) and a statistical bound T = Omega(sigma^2 d / eps^2), and the sharp product-form bound T = Omega((sigma^2 d / eps^2) max{1, d/B}). The product form is also unconditional: a B-bit transcript carries at most O(TB / sigma^2) of Fisher trace about the mean, so bits rather than dimension limit the recoverable information, and combined with the multivariate van Trees inequality this gives the bound directly, without bounded-likelihood-ratio truncation. We give a near-matching achievability result with exact per-round bit accounting under a bounded-dynamic-range oracle, tight up to a logarithmic factor; the lower bound is for truly Gaussian (unbounded) gradients, and closing this oracle gap is left open. A sequential rate-distortion perspective extends the reduction to correlated and drifting oracles and corrects an earlier conjecture: positive noise correlation raises the bound by (1+rho)/(1-rho) rather than relaxing it. The bounds give an information-theoretic baseline for any low-bit gradient path, not an optimality claim about deployed FP4 systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。