一种可适配多种模型的后处理方法,有效缓解概念漂移问题。
NOMADD: Numerical Optimization of Models Adapting to Data Drift

- 基于参数演化轨迹建模,通过低秩压缩与正则化外推实现漂移预测
- 在18个数据集上提升所有基线模型性能,推理速度远超现有最优方案
- 适合对精度、延迟和模型大小有严格约束的应用场景
表格模型在特征分布随时间变化或特征与目标变量关系改变时性能下降,即数据漂移与概念漂移。此类问题难以实时缓解,因标注数据可能无法即时获取,或重新训练不切实际。现有工具多针对特定神经网络架构,且仅限于训练阶段优化。本文提出一种适用于树模型、神经网络及表格基础模型等多种架构的后处理方法,解决概念漂移问题。算法分别在各标注训练期拟合基线模型,测量其参数演变相对于全周期锚定模型的变化,通过低秩分解压缩变化,并以阻尼正则化方式外推潜变量。在18数据集的Drift-Resilient TabPFN基准上,该方法在原基准协议与指标下显著提升所有基线家族表现,性能媲美当前最优的Drift-Resilient TabPFN,仅需数秒训练;而后者需在百万合成数据集上预训练约1300 GPU小时,且推理速度慢一个数量级以上。讨论部分展望了该方法向其他模态扩展的可能性。
原文摘要 · Abstract (English)
Tabular model performance degrades when feature distributions change over time or the relationship between features and outcome variables change over time, known as data drift and concept drift, respectively. These issues are challenging to mitigate in real time because labeled data may not be immediately available, or re-training a model could be impractical. While tools exist to reduce drift, they are typically bespoke to neural network architectures and adapt how models are trained. In this paper, we offer an alternative post-hoc method to reduce concept drift, which is applicable to a variety of models, from trees to neural networks to tabular foundation models. This new tool is especially useful when constraints, such as high model accuracy, bounded inference time, or model size requires users to choose between different models for their specific use-cases. Our algorithm fits the base model separately on each labeled training period, measures how its parameters evolve against a single anchor model pooled over all of those periods, compresses those changes with a low-rank factorization, and extrapolates each latent factor forward with a damped, regularized forecast. On the 18-dataset Drift-Resilient TabPFN benchmark, evaluated under that benchmark's own protocol and metric, the extrapolation improves every base family it is applied to, and achieves performance competitive with the state-of-the-art Drift-Resilient TabPFN with seconds of training. In contrast, Drift-Resilient TabPFN requires pre-training on millions of synthetic datasets over approximately 1,300 GPU-hours, and is orders of magnitude slower in inference (depending on the model). In the discussion, we explore the promise and challenges of extending this tool to other modalities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。