通过动态调整选数据量,让模型训练更高效且稳定。
Beyond What to Select: A Plug-and-play Oscillatory Data-Volume Scheduling for Efficient Model Training

- 在训练中周期性切换选数据的多少,利用选择带来的正则化效果。
- 在ImageNet-1k上节省50%训练成本,准确率还更高;大模型微调提速2倍以上。
- 不依赖具体选数据方法,可直接加到现有训练流程中使用。
数据选择通过筛选有代表性的训练样本加速模型训练,同时保持性能。然而,现有方法主要关注如何判断选哪些样本(即‘选什么’),通常固定训练过程中选择的数据比例。因此,这些方法在样本身份上是动态的,但在数据量上是静态的。本文从优化视角重新审视数据选择,发现选中数据的训练会引入由当前选择比例调控的隐式正则化效应。这揭示了一个关键权衡:较低的选择比例能增强选择诱导的正则化,而较高的比例则保障数据覆盖和优化保真度。基于此,我们提出PODS——一种即插即用的振荡式数据量调度框架。不同于设计新的样本评分标准,PODS作为一个轻量模块,在目标选择比例下动态调度每阶段选择的数据量。它在低比例正则化阶段与高比例恢复阶段之间交替,以利用选择诱导的正则化,同时不牺牲优化稳定性。由于其轻量化、比例级和任务无关的设计,PODS兼容现有的静态与动态选择方法,并广泛适用于各类训练范式。跨多种数据集、模型架构和任务的实验表明,PODS始终提升效率与泛化之间的平衡,例如在ImageNet-1k上将训练成本降低50%且准确率提升,同时在大模型指令微调中实现超过2倍的加速,且无性能损失。
原文摘要 · Abstract (English)
Data selection accelerates training by identifying representative training data while preserving model performance. However, existing methods mainly focus on designing sample-importance criteria, i.e., deciding what to select, while typically fixing the selected data volume as the target ratio throughout training. Thus, they are often dynamic in sample identity but static in data volume. In this work, we revisit data selection from an optimization perspective and show that selected-data training induces an implicit regularization effect modulated by the instantaneous selection ratio. This reveals a key trade-off: lower ratios amplify selection-induced regularization, whereas higher ratios preserve data coverage and optimization fidelity. Motivated by this insight, we propose PODS, a Plug-and-play Oscillatory Data-volume Scheduling framework. Rather than introducing another sample-scoring metric, PODS serves as a lightweight module that dynamically schedules how much data to select over training. Under the target selection ratio, PODS alternates between low-ratio regularization phases and high-ratio recovery phases to exploit selection-induced regularization without sacrificing optimization stability. With its lightweight, ratio-level, and task-agnostic design, PODS is compatible with existing static and dynamic selection methods and broadly applicable across training paradigms. Experiments across various datasets, architectures, and tasks show that PODS consistently improves the efficiency-generalization trade-off, e.g., reducing ImageNet-1k training cost by 50% with improved accuracy and accelerating LLM instruction tuning by over 2x without performance degradation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。