arXiv:2509.17304cs.LG2025-09被引 2

提出新算法SPRINT,显著提升非凸场景下机器学习模型的稳定收敛速度。

SPRINT: Stochastic Performative Prediction With Variance Reduction

  • 引入方差缩减技术,改进随机梯度估计以应对数据分布变化。
  • 收敛速度达O(1/T),误差范围与梯度方差无关。
  • 适合在数据分布受模型影响的动态场景中应用,如在线推荐系统。

Performative prediction (PP) 是一种机器学习优化框架,其特点是模型部署会改变训练数据的分布。与传统固定数据的机器学习相比,寻找稳定的可执行预测(SPS)解更具挑战性。尽管已有研究采用重复梯度下降(RGD)和贪婪随机梯度下降(SGD-GD)等方法,但多数假设损失函数为强凸。最近工作证明,在平滑非凸损失下,SGD-GD 可实现 $O(1/ ext{√}T)$ 的收敛率。然而,该结果依赖于有界方差假设,且存在不随 $T$ 消失的误差邻域,其大小与梯度方差成正比。为突破此限制,本文提出新的方差缩减算法 SPRINT,首次在非凸设置下实现 $O(1/T)$ 的收敛率,且误差邻域独立于梯度方差。在多个真实数据集上的实验表明,SPRINT 在收敛速度与稳定性方面均优于 SGD-GD。

原文摘要 · Abstract (English)

Performative prediction (PP) is an algorithmic framework for optimizing machine learning (ML) models where the model's deployment affects the distribution of the data it is trained on. Compared to traditional ML with fixed data, designing algorithms in PP converging to a stable point -- known as a stationary performative stable (SPS) solution -- is more challenging than the counterpart in conventional ML tasks due to the model-induced distribution shifts. While considerable efforts have been made to find SPS solutions using methods such as repeated gradient descent (RGD) and greedy stochastic gradient descent (SGD-GD), most prior studies assumed a strongly convex loss until a recent work established $O(1/\sqrt{T})$ convergence of SGD-GD to SPS solutions under smooth, non-convex losses. However, this latest progress is still based on the restricted bounded variance assumption in stochastic gradient estimates and yields convergence bounds with a non-vanishing error neighborhood that scales with the variance. This limitation motivates us to improve convergence rates and reduce error in stochastic optimization for PP, particularly in non-convex settings. Thus, we propose a new algorithm called stochastic performative prediction with variance reduction (SPRINT) and establish its convergence to an SPS solution at a rate of $O(1/T)$. Notably, the resulting error neighborhood is independent of the variance of the stochastic gradients. Experiments on multiple real datasets with non-convex models demonstrate that SPRINT outperforms SGD-GD in both convergence rate and stability.

非凸优化方差缩减自适应学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。