arXiv:2411.15616cs.LG2024-11中稿 · CODS-COMAD 2024被引 1

通过自适应数据分段,同时应对数据分布和概念漂移,提升模型长期稳定性。

A Scalable Approach to Covariate and Concept Drift Management via Adaptive Data Segmentation

  • 基于动态分段识别匹配测试数据的最优训练批次
  • 在真实与合成数据上提升准确率并降低延迟和成本
  • 适合需要持续更新的大型机器学习系统

在众多实际应用中,持续机器学习系统至关重要,但易受数据漂移影响——历史训练数据与未来测试数据之间的差异会导致性能显著下降和运营效率降低。传统漂移适应方法通常采用集成技术更新模型,常丢弃漂移后的历史数据,且主要关注协变量漂移或概念漂移之一。此类方法存在资源消耗高、无法有效管理多种漂移类型、忽视历史数据价值等问题。本文提出一种融合数据驱动策略与自适应漂移管理的先进框架,通过复杂的数据分段技术识别能准确反映测试数据模式的最优数据批次,并用于测试数据上的训练,确保模型随时间保持相关性和准确性。该方案结合数据分段与可扩展漂移管理优势,显著降低资源消耗与计算开销,实现大规模部署下的高鲁棒性与高效性。在分类任务的真实与合成数据集上的实验表明,本方法在提升模型准确率的同时,有效降低运营成本与延迟,克服了现有方法的低效问题,提供了一种稳健、可扩展且实用的解决方案。

原文摘要 · Abstract (English)

In many real-world applications, continuous machine learning (ML) systems are crucial but prone to data drift, a phenomenon where discrepancies between historical training data and future test data lead to significant performance degradation and operational inefficiencies. Traditional drift adaptation methods typically update models using ensemble techniques, often discarding drifted historical data, and focus primarily on either covariate drift or concept drift. These methods face issues such as high resource demands, inability to manage all types of drifts effectively, and neglecting the valuable context that historical data can provide. We contend that explicitly incorporating drifted data into the model training process significantly enhances model accuracy and robustness. This paper introduces an advanced framework that integrates the strengths of data-centric approaches with adaptive management of both covariate and concept drift in a scalable and efficient manner. Our framework employs sophisticated data segmentation techniques to identify optimal data batches that accurately reflect test data patterns. These data batches are then utilized for training on test data, ensuring that the models remain relevant and accurate over time. By leveraging the advantages of both data segmentation and scalable drift management, our solution ensures robust model accuracy and operational efficiency in large-scale ML deployments. It also minimizes resource consumption and computational overhead by selecting and utilizing relevant data subsets, leading to significant cost savings. Experimental results on classification task on real-world and synthetic datasets show our approach improves model accuracy while reducing operational costs and latency. This practical solution overcomes inefficiencies in current methods, providing a robust, adaptable, and scalable approach.

数据漂移自适应学习可扩展性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。