用安全的大学习率解决在线模型选择的适应延迟问题
Agile Online Model Selection: Resolving Adaptation Lag via Safeguarded Large Learning Rates
- 提出后置惩罚机制，动态监控并排除导致过大学习损失的更新
- 学习率可达Θ(T)，使适应延迟从数百轮降至几轮
- 适合需要快速响应分布变化的实时系统场景
在非平稳环境中保持预测准确性需要在线模型选择以自主适应未知的数据分布漂移。然而，现有免调参算法在鲁棒性与敏捷性之间存在根本权衡：为保证动态后悔边界，必须将学习率限制在小常数（如O(1)），这导致突发变化时显著的适应延迟。为此，我们提出一种新型乐观在线镜下降算法，采用高达Θ(T)的安全大学习率，其中T为轮次数。核心技术贡献是后置惩罚机制，可动态监测不稳定的更新，并剔除导致过度后悔的学习率，无需预先施加严格约束。我们证明累积惩罚为O(log T)，使算法在最坏情况下逼近最优保证，同时在良性情形下实现更优速率。在三个合成数据集和十一个真实世界数据集上的实证评估表明，该方法将适应延迟从数百轮降至几轮，持续优于免调参基线。
原文摘要 · Abstract (English)
Maintaining predictive accuracy in non-stationary environments requires online model selection to adapt autonomously to unknown distribution shifts. However, existing tuning-free algorithms face a fundamental trade-off between robustness and agility. Specifically, to ensure dynamic regret bounds, they must restrict learning rates to small constants (e.g., $O(1)$). This restriction inevitably causes significant adaptation lag during abrupt changes. To resolve this, we propose a novel optimistic online mirror descent that utilizes safeguarded large learning rates up to $Θ(T)$, where $T$ is the number of rounds. Our key technical contribution is a post-hoc penalty mechanism that dynamically monitors unstable updates and excludes learning rates incurring excessive regret, eliminating the need for restrictive a priori constraints. We show that the cumulative penalty remains $O(\log T)$, allowing our algorithm to match near-optimal worst-case guarantees while achieving superior rates in benign cases. Empirical evaluations on three synthetic and eleven diverse real-world datasets demonstrate that our approach reduces the adaptation lag from hundreds of rounds to a few rounds, consistently outperforming tuning-free baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。