arXiv:2605.10673cs.LG2026-05

提出新型查询几何方法,解决低比特零阶优化中的量化误差问题。

Compander-Aligned Query Geometry for Quantized Zeroth-Order Optimization

论文配图:Compander-Aligned Query Geometry for Quantized Zeroth-Order Optimization
图 1 · 摘自论文原文
  • 在压缩映射空间中构造对齐查询,避免量化带来的方向偏差。
  • 理论证明可将查询时残差降为零,提升优化稳定性。
  • 适用于低精度模型微调,尤其适合资源受限场景下的高效优化。

低比特前向评估为内存高效的零阶(ZO)自适应提供了一条可行路径:优化器仅需标量损失,模型可在接近部署精度下查询。但量化后的ZO查询并非连续有限差分后简单舍入。查询选择端点,低精度引擎进行舍入,损失差值沿舍入后的弦测量。对于非均匀压缩量化器,码本无法准确预测ZO行为:固定权重空间半径可能在密集单元坍缩、过度跨越稀疏单元,或使舍入弦对应未舍入的更新方向。我们识别出缺失的关键是查询几何,并将标量非均匀量化建模为 $Q = ϕ^{-1} igcirc U igcirc ϕ$。CAQ-ZO(压缩对齐查询)在 $z = ϕ(x)$ 空间中形成单步网格的Rademacher模板 $z \pm Δr$,通过 $ϕ^{-1}$ 映射回原空间并更新。理论证明了网格跨度不匹配,分解了端点舍入估计残差,并给出了稳态边界:通用离格查询保留 $Δ^2/μ^2$ 残差通道,而CAQ-ZO使查询时残差精确为零。合成实验验证了该通道,且在相同量化器与评估预算下,匹配的NF4 Qwen/Llama微调表明CAQ-ZO优于基准模型。

原文摘要 · Abstract (English)

Low-bit forward evaluation is an attractive route to memory-efficient zeroth-order (ZO) adaptation: the optimizer needs only scalar losses, and the model can be queried near deployment precision. The obstacle is that a quantized ZO query is not a continuous finite difference followed by harmless storage rounding. The query chooses endpoints, the low-precision engine rounds them, and the loss difference is measured along the rounded chord. For nonuniform companding quantizers, this makes the codebook insufficient to predict ZO behavior: a fixed weight-space radius can collapse in dense cells, over-span sparse cells, or assign a rounded chord to an unrounded update direction. We identify the missing object as query geometry and model scalar nonuniform quantization as $Q = ϕ^{-1} \circ U \circ ϕ$. CAQ-ZO (Compander-Aligned Queries for Zeroth-Order Optimization) forms one-grid-step Rademacher stencils $z \pm Δr$ in $z = ϕ(x)$, maps endpoints back through $ϕ^{-1}$, and updates in $z$. Our theory proves the grid-span mismatch, decomposes endpoint-rounding estimator residuals, and gives stationarity bounds in which generic off-grid queries retain a $Δ^2/μ^2$ residual channel while CAQ-ZO makes the query-time residual exactly zero. Synthetic experiments isolate this channel, and matched NF4 Qwen/Llama fine-tuning shows that CAQ-ZO improves the trained NF4 baseline under the same quantizer and evaluation budget.

零阶优化量化压缩映射高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。