提出改进版在线学习算法,能快速适应动态数据流。
Momentum LMS Theory beyond Stationarity: Stability, Tracking, and Regret
- 引入动量机制增强在线学习稳定性
- 理论证明算法在非平稳环境下跟踪误差可控
- 适合实时处理数据流的工程场景
在大规模数据处理中,数据常以复杂系统生成的时变流形式出现,分布和参数随时间漂移,打破传统i.i.d.假设。为此,需设计可实时更新、无需昂贵重训练的算法,要求单次遍历样本,且计算与内存开销不随数据流长度增长。本文研究动量最小均方(MLMS)算法作为自适应识别工具,利用其计算简单性和在线处理能力。理论上,推导了在时变随机线性系统下MLMS的跟踪性能与遗憾边界,不同于经典LMS的一阶随机向量差分方程,MLMS因动量引入额外动态状态,导致二阶时变随机向量差分方程,其稳定性分析依赖于更复杂的随机矩阵乘积,带来显著挑战。合成与真实数据流实验表明,MLMS实现快速适应与鲁棒跟踪,与理论结果一致,尤其在非平稳环境下表现突出,展现出在现代流式与在线学习中的应用潜力。
原文摘要 · Abstract (English)
In large-scale data processing scenarios, data often arrive in sequential streams generated by complex systems that exhibit drifting distributions and time-varying system parameters. This nonstationarity challenges theoretical analysis, as it violates classical assumptions of i.i.d. (independent and identically distributed) samples, necessitating algorithms capable of real-time updates without expensive retraining. An effective approach should process each sample in a single pass, while maintaining computational and memory complexities independent of the data stream length. Motivated by these challenges, this paper investigates the Momentum Least Mean Squares (MLMS) algorithm as an adaptive identification tool, leveraging its computational simplicity and online processing capabilities. Theoretically, we derive tracking performance and regret bounds for the MLMS in time-varying stochastic linear systems under various practical conditions. Unlike classical LMS, whose stability can be characterized by first-order random vector difference equations, MLMS introduces an additional dynamical state due to momentum, leading to second-order time-varying random vector difference equations whose stability analysis hinges on more complicated products of random matrices, which poses a substantially challenging problem to resolve. Experiments on synthetic and real-world data streams demonstrate that MLMS achieves rapid adaptation and robust tracking, in agreement with our theoretical results especially in nonstationary settings, highlighting its promise for modern streaming and online learning applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。