arXiv:2512.01881cs.LGcs.AI2025-12

动态调节梯度更新精度,让优化器自动在探索与精修间切换。

Unifying Sign and Magnitude for Optimizing Deep Vision Networks via ThermoLion

  • 根据信号噪声比自动切换低比特探索和高精度精修模式
  • 在12个视觉数据集上收敛更快、准确率更高
  • 适合追求高效训练的计算机视觉研究者

深度视觉模型的训练本质上是在高维随机噪声中恢复信号的问题。现有优化方法在信息通道容量上采取静态权衡:基于幅值的方法(如AdamW)假设梯度范数是可靠的曲率信号,在平滑区域可实现高精度,但在崎岖非凸空间会放大噪声;而基于符号的方法(如Lion)对梯度进行1比特量化,虽具鲁棒性,却损失了精细下降信息。本文提出,最优收敛需动态调节更新比特率。我们引入面向视觉任务的ThermoLion框架,利用局部信噪比门控机制,自主实现参数在“低比特探索”与“高精度利用”间的切换;同时提出动量对齐机制,检测历史漂移与瞬时梯度间的建设性干涉,加速稳定轨迹下的收敛。在12个不同视觉数据集(包括CIFAR、SVHN、GTSRB)上的实证表明,ThermoLion在收敛速度与最终精度上均超越AdamW、Lion等顶尖优化器。

原文摘要 · Abstract (English)

The training of deep vision models is fundamentally a signal recovery problem amidst high-dimensional stochastic noise. Current optimization paradigms impose a static compromise on information channel capacity. For instance, magnitude-based methods, such as AdamW, operate on the assumption that gradient norms are high-fidelity curvature signals. While this allows for precision in smooth regimes, it leads to catastrophic noise amplification when applied to rugged, non-convex landscapes. Conversely, sign-based methods (e.g., Lion) perform a radical 1-bit quantization of the gradient, which aims to provide robust regularization at the cost of discarding fine-grained descent information. We propose that optimal convergence requires neither static prior, but rather a dynamic modulation of the update bitrate. We introduce ThermoLion, a vision-centric framework that utilizes local Signal-to-Noise Ratio (SNR) gating to autonomously transition parameters between a "low-bit" exploration phase and a "high-precision" exploitation phase. Furthermore, we introduce a Momentum Alignment mechanism that detects constructive interference between historical drift and instantaneous gradients to accelerate convergence during stable trajectories. Empirical benchmarks across 12 diverse vision datasets (including CIFAR, SVHN, and GTSRB) demonstrate that ThermoLion surpasses state-of-the-art optimizers, such as AdamW and Lion, in convergence speed and terminal accuracy.

优化器视觉模型自适应训练加速

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。