arXiv:2607.27273cs.LG2026-07

动态调整训练数据组织,让模型更高效地学习不同难度问题。

SDO: Structure-Aware Data Organization for Efficient LLM Post-Training

论文配图:SDO: Structure-Aware Data Organization for Efficient LLM Post-Training
图 1 · 摘自论文原文
  • 根据数据在特征空间的结构,动态分组和安排训练样本
  • 早期到中期收敛速度提升显著,各类问题准确率更均衡
  • 无需重新训练模型,可直接用于SFT、DPO等场景

大语言模型后训练成本高昂,现有优化多聚焦于选择有效样本或设计训练节奏。但数据组织通常被视为静态预处理:基于嵌入的分组方法在训练前构建固定分区,无法适应优化过程中的样本暴露变化。结果是所有样本获得相似曝光,部分样本重复更新而其他样本未充分优化。为此,我们提出SDO(结构感知数据组织)框架,采用暴露驱动反馈机制,依据表示空间结构动态组织小批量构成与样本曝光。SDO在冻结外部嵌入的基础上逐轮运行,避免模型预热开销:每轮内,通过KNN邻域遍历形成局部性一致的小批量;跨轮次间,记录每个样本参与度,降低过度曝光样本的采样概率,以维持长期覆盖。在SFT、DPO和GRPO任务中,SDO加速收敛,早期至中期提升最明显,生成更连贯梯度,实现各类问题准确率更均衡,且不永久剔除任何训练样本。

原文摘要 · Abstract (English)

Post-training of large language models is expensive, and existing efficiency improvements mainly focus on selecting informative samples or designing training schedules. However, data organization itself is usually treated as a static preprocessing step: embedding-based grouping methods construct fixed partitions before training and cannot adapt to the evolving sample exposure during optimization. As a result, all samples receive similar exposure despite their different optimization needs, leading to redundant updates for some samples while leaving others under-optimized. To address this problem, we propose SDO (Structure-Aware Data Organization), a plug-and-play data organization framework with an exposure-driven feedback mechanism that organizes mini-batch composition and sample exposure according to representation-space structure. SDO operates epoch by epoch on frozen external embeddings, avoiding model warm-up training overhead: within each epoch, locality-aware batching forms coherent mini-batches via KNN neighborhood traversal; across epochs, exposure-balanced scheduling records per-sample participation and reduces the sampling probability of over-exposed samples to preserve long-term coverage. Across SFT, DPO, and GRPO, SDO accelerates convergence, with the largest gains observed in the early-to-mid phase, producing more coherent gradients and more balanced accuracy across question types without permanently excluding training samples.

大模型训练数据组织高效微调结构感知

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。