用强化学习选数据，5%样本效果超全量数据。
RL-Guided Data Selection for Language Model Finetuning
- 将选数据转为可解的马尔可夫决策过程，用强化学习学策略。
- 仅用5%数据训练，准确率比全量数据高10.8点，提速两倍。
- 适合资源有限但追求高精度的模型微调场景。
大语言模型微调中的数据选择可视为一个预算受限的优化问题：在严格的数据预算下最大化下游性能。该问题通常难以求解，现有近似方法多基于预训练，难以迁移到微调场景。本文将其重新建模为可解的马尔可夫决策过程（MDP），并使用多种强化学习（RL）方法训练智能体，学习由高效代理模型提供的奖励信号引导的数据选择策略。在四个数据集上，使用我们方法选出的5%子集进行训练，其性能达到或超过在全量数据上微调的效果，最高提升达10.8个准确率点，同时将实际训练时间减少最多2倍，凸显了强化学习引导数据选择的巨大潜力。
原文摘要 · Abstract (English)
Data selection for finetuning Large Language Models (LLMs) can be framed as a budget-constrained optimization problem: maximizing a model's downstream performance under a strict training data budget. Solving this problem is generally intractable, and existing approximate approaches are pretraining-oriented and transfer poorly to the fine-tuning setting. We reformulate this problem as a tractable Markov Decision Process (MDP) and train agents using various Reinforcement Learning (RL) methods to learn optimal data selection policies, guided by an efficient, proxy-model-based reward signal. Across four datasets, training on a $5\%$ subset selected by our approach matches or outperforms fine-tuning on the full dataset by up to $10.8$ accuracy points, while cutting wall-clock training time by up to $2 \times$, highlighting the promise of RL-guided data selection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。