针对动态数据训练,提出两阶段高效调参方法,成本降10倍。
Efficient Hyperparameter Search for Non-Stationary Model Training
- 分两阶段:先快速筛选优质配置,再全量训练
- 在Criteo 1TB数据上节省超10倍调参成本
- 适合大规模在线学习系统,如推荐与广告
在线学习是推荐和广告系统的核心,模型需持续适应变化的数据分布。此类系统的训练成本极高,超参搜索更会放大这一开销。本文提出一种两阶段范式:(1) 高效识别最具潜力的超参配置;(2) 仅对选中候选进行完整训练。核心思想是第一阶段侧重精准识别而非性能峰值,从而实现大幅降本。我们设计了新型数据压缩与预测策略,专门应对序列性、非平稳数据带来的挑战,弥补传统超参优化的不足。通过双重验证:首先在最大公开基准Criteo 1TB数据集上测试,其次在工业级广告系统(规模比基准大两个数量级)部署。结果表明,该方法在公共基准上将总超参搜索成本降低最多达10倍,并在真实工业场景中实现显著且可验证的效率提升。
原文摘要 · Abstract (English)
Online learning is the cornerstone of applications like recommendation and advertising systems, where models continuously adapt to shifting data distributions. Model training for such systems is remarkably expensive, a cost that multiplies during hyperparameter search. We introduce a two-stage paradigm to reduce this cost: (1) efficiently identifying the most promising configurations, and then (2) training only these selected candidates to their full potential. Our core insight is that focusing on accurate identification in the first stage, rather than achieving peak performance, allows for aggressive cost-saving measures. We develop novel data reduction and prediction strategies that specifically overcome the challenges of sequential, non-stationary data not addressed by conventional hyperparameter optimization. We validate our framework's effectiveness through a dual evaluation: first on the Criteo 1TB dataset, the largest suitable public benchmark, and second on an industrial advertising system operating at a scale two orders of magnitude larger. Our methods reduce the total hyperparameter search cost by up to 10$\times$ on the public benchmark and deliver significant, validated efficiency gains in the industrial setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。