arXiv:2605.07063cs.LGcs.AI2026-05

用数据正则化思路提升大模型微调效果,避免过拟合稀缺数据。

Dr. Post-Training: A Data Regularization Perspective on LLM Post-Training

  • 将通用数据视为正则项,动态约束目标数据的更新方向。
  • 在SFT、RLHF、RLVR任务中均超越现有数据筛选方法。
  • 适合需要高效利用少量高质量数据的LLM微调场景。

数据选择方法应对大模型微调中的核心挑战:如何有效利用稀缺但高保真目标数据,同时结合大量不完美对齐的通用训练数据。本文突破传统数据选择视角,提出Dr. Post-Training(数据正则化微调)框架,将通用数据重新构想为一种由数据诱导的正则化项,防止模型过度拟合稀缺目标数据,而非单纯作为可选数据池。具体而言,每一步训练中,利用通用数据构建可行的模型更新方向集合,并将目标数据指定的更新方向投影到该集合上。标准训练与现有数据选择方法均为特定正则化策略下的特例,对应于偏差-方差谱上不同正则强度的位置。基于此视角,我们提出一系列方法,提供更丰富的设计空间与灵活的偏差-方差权衡。针对大模型规模应用,引入精细系统优化,实现极低开销。跨SFT、RLHF、RLVR的广泛实验表明,所提方法持续优于当前最优数据选择基线;系统基准测试也验证了其高效性。

原文摘要 · Abstract (English)

Data selection methods address a critical challenge in LLM post-training: effectively leveraging scarce, high-fidelity target data alongside abundant but imperfectly aligned general training data. In this work, we move beyond the data-selection framing and introduce Dr. Post-Training (Data-Regularized Post-Training), a novel framework that reconceptualizes general training data as a data-induced regularizer that prevents overfitting to the scarce target objective, rather than serving as a pool for selection. Specifically, our framework proposes that at each training step, construct a feasible set of model update directions using the general training data, and project the model update direction specified by the scarce target data onto that feasible set. Standard training and existing data selection methods arise as special cases with different choices of the data-induced regularizer, and these methods correspond to different points on a bias--variance spectrum with different regularization strength. Building on this view, we propose a family of methods offering a richer design space and more flexible bias--variance tradeoffs. For practical LLM-scale use, we introduce careful system optimizations that realize these methods with minimal overhead. Extensive experiments across SFT, RLHF, and RLVR show that our methods consistently outperform state-of-the-art data selection baselines, and system benchmarks confirm their efficiency.

大模型微调数据正则化强化学习高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。