arXiv:2509.04713cs.LG2025-09被引 1

提出新型优化器调度方法,让训练更高效、收敛更快。

Natural Spectral Fusion: p-Exponent Cyclic Scheduling and Early Decision-Boundary Alignment in First-Order Optimization

  • 用周期性重加权频段控制优化过程,不改模型和数据
  • 负指数可增强高频信息,实现早期决策边界对齐
  • 相同设置下测试误差更低,部分任务只需1/4训练成本

谱行为在机器学习中被广泛讨论,但优化器自身的谱偏差仍不明确。我们指出一阶优化器存在内在频率偏好,显著影响优化路径。为此提出自然谱融合(NSF):将训练重构为可控的谱覆盖与信息融合,而非单纯缩放步长。NSF包含两个核心原则:将优化器视为谱控制器,动态平衡低频与高频信息;通过周期性重加权频段实现,成本极低,无需修改模型、数据或训练流程。通过扩展二阶矩项的p-指数形式,支持正负指数,结合周期调度实现。理论与实验表明,自适应方法侧重低频,SGD近似中性,负指数可放大高频信息。周期调度扩大谱覆盖范围,提升跨频段融合能力,并诱导早期决策边界对齐——准确率提升时损失仍高。在多个基准上,使用相同学习率策略和固定超参,p-指数周期调度持续降低测试误差,某些任务仅需四分之一训练成本即达到基线精度。总体而言,NSF揭示了优化器作为主动谱控制器的角色,提供统一、可控且高效的逐阶优化框架。

原文摘要 · Abstract (English)

Spectral behaviors have been widely discussed in machine learning, yet the optimizer's own spectral bias remains unclear. We argue that first-order optimizers exhibit an intrinsic frequency preference that significantly reshapes the optimization path. To address this, we propose Natural Spectral Fusion (NSF): reframing training as controllable spectral coverage and information fusion rather than merely scaling step sizes. NSF has two core principles: treating the optimizer as a spectral controller that dynamically balances low- and high-frequency information; and periodically reweighting frequency bands at negligible cost, without modifying the model, data, or training pipeline. We realize NSF via a p-exponent extension of the second-moment term, enabling both positive and negative exponents, and implement it through cyclic scheduling. Theory and experiments show that adaptive methods emphasize low frequencies, SGD is near-neutral, and negative exponents amplify high-frequency information. Cyclic scheduling broadens spectral coverage, improves cross-band fusion, and induces early decision-boundary alignment, where accuracy improves even while loss remains high. Across multiple benchmarks, with identical learning-rate strategies and fixed hyperparameters, p-exponent cyclic scheduling consistently reduces test error and demonstrates distinct convergence behavior; on some tasks, it matches baseline accuracy with only one-quarter of the training cost. Overall, NSF reveals the optimizer's role as an active spectral controller and provides a unified, controllable, and efficient framework for first-order optimization.

优化算法谱分析高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。