arXiv:2608.19488cs.LG2026-08

实证研究发现,增量学习比重训策略更重要。

When to Retrain: An Empirical Study of Retraining Policies for Streaming ML Under Concept Drift, Budget, and Latency Constraints

  • 对比三种重训策略与无重训基线,考察预算与延迟影响。
  • 无增量更新时,策略差异导致准确率相差15%-55%。
  • 适合关注在线学习系统稳定性的工程师和研究者。

生产环境中的机器学习系统会因概念漂移而性能下降,但从业者缺乏关于何时重训的系统性指导。重训成本高,预算有限,且模型重训后无法立即生效:训练与部署延迟导致旧模型持续服务预测,数据却仍在流动。本文通过受控实验,评估三种实用的模型刷新策略(周期性重训、误差阈值触发、统计漂移触发的ADWIN)与不重训基线的性能,统一建模重训预算与训练加部署延迟。在3,933次实验中,覆盖三种漂移模式、三种预算水平、最多五种延迟水平、三个数据集及两种学习模式,结果表明:最关键的决策不是重训策略,而是部署模型是否支持增量学习。对于线性在线学习器且标签即时可用的情况,采用样本级增量更新后,所有策略与无重训基线在54组对比中均无显著差异,即使在极端延迟下也如此。若无增量更新,策略选择导致漂移后准确率相差15%-55%,周期性重训在突变与渐变漂移下显著优于反应式策略,而反应式策略仅在周期性漂移中占优。本文还揭示了反应式策略的系统性失败模式,以及延迟-预算队列交互效应使有效重训预算减半的现象,并公开完整模拟器、数据管道与每轮实验结果以保障可复现性。

原文摘要 · Abstract (English)

Production machine learning systems degrade under concept drift, yet practitioners have little principled guidance on when to retrain. Retraining is costly, retraining budgets are finite, and a retrained model does not take effect instantly: training and deployment latency leave a stale model serving predictions while the data continues to move. We present a controlled empirical study of three practical model-refresh policies (periodic retraining, error-threshold triggering, and statistical drift-triggered retraining with ADWIN) against a no-retrain baseline, evaluated under a unified system model that makes retraining budgets and training-plus-deployment latency explicit. Across 3,933 experiment runs spanning three drift regimes, three budget levels, up to five latency levels, three datasets, and two learning modes, we find that the single most consequential design decision is not the retraining policy but whether the deployed model learns incrementally. With per-sample incremental updates, and for the linear online learner with immediate labels studied here, no policy differs from the no-retrain baseline by a practically significant margin in any of 54 paired comparisons, even at extreme latency. Without incremental updates, policy choice separates outcomes by 15-55 percentage points of post-drift accuracy, and simple periodic retraining significantly outperforms both reactive policies under abrupt and gradual drift, while reactive policies retain an advantage only under recurring drift. We document systematic failure modes of reactive policies and a latency-budget queueing interaction that silently halves effective retraining budgets, and release the full simulator, dataset pipelines, and per-run artifacts for reproducibility.

在线学习概念漂移重训策略增量学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。