用强化学习改造训练数据，让大模型推理更通用。
Learning to Adapt SFT Data for Better Reasoning Generalization
- 训练映射模型转化原始数据，匹配目标模型分布。
- 在多个模型上提升推理泛化能力，效率高于直接强化学习。
- 适合需要提升推理通用性的大模型微调场景。
大语言模型（LLMs）的性能提升很大程度依赖后训练。其中，监督微调（SFT）通过外部数据提供密集监督，实现高效训练。然而，当原始数据分布与目标模型分布不一致时，直接使用专家数据会损害泛化能力。本文提出数据适应推理微调（DART），将固定但可能分布不匹配的SFT数据使用问题建模为演示转换的优化问题。DART通过强化学习训练一个映射模型，将原始SFT数据转化为更符合目标模型分布和学习偏好的适配数据。这些转换后的数据用于SFT，使目标模型能更好利用外部监督。在多个模型和数据集上的实验表明，DART提升了泛化能力，训练效率优于直接强化学习，并帮助模型超越标准SFT。代码已公开于 https://anonymous.4open.science/r/DART525E50D。
原文摘要 · Abstract (English)
Large language models (LLMs) have achieved remarkable progress, with post-training playing a crucial role in enhancing their reasoning capabilities. Among post-training paradigms, supervised fine-tuning (SFT) is widely used: it leverages external data to provide dense supervision and enables efficient training. However, directly fine-tuning on expert data can hurt generalization when the data distribution is mismatched with the target model's own distribution. In this work, we propose Data Adaptation for Reasoning Tuning (DART), which formulates the use of a fixed, potentially distributionally misaligned SFT dataset as an optimization problem over demonstration transformations. DART trains a mapper model with reinforcement learning to convert original SFT data into model-adapted supervision that better matches the target model's distribution and learning preferences. The transformed data are then used for SFT, allowing the target model to better exploit external supervision. Experiments across multiple models and datasets show that DART improves generalization, achieves higher training efficiency than direct RL, and helps models surpass standard SFT. Our code is available at https://anonymous.4open.science/r/DART525E50D.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。