arXiv:2504.07282cs.CL2025-04EMNLP被引 2

用强化学习动态选指令,让大模型训练更高效精准。

RAISE: Reinforced Adaptive Instruction Selection For Large Language Models

  • 将选指令变成序列决策问题,用强化学习动态优化每步选择。
  • 仅用1%训练步数就达到全量数据训练效果,效率大幅提升。
  • 针对具体任务优化,可解释性强,适合需要高效微调的场景。

在大语言模型的指令微调中,少量高质量指令优于大量低质量指令。现有方法多依赖启发式质量指标进行数据筛选,且仅在训练前固定选择,未能动态优化整个微调过程,导致性能受限。为此,本文提出一种动态、任务目标驱动的指令选择框架RAISE(Reinforced Adaptive Instruction SElection),将整个微调过程纳入优化,基于每条指令对模型性能提升的预期影响,在每一步动态选择最优指令。通过将动态指令选择建模为序列决策问题,并使用强化学习训练选择策略,RAISE具备强可解释性和任务自适应能力。大量实验表明,相比其他方法,RAISE表现更优;尤其值得注意的是,其仅需更新1%的训练步数即可达到全数据训练的效果,充分验证了其高效性与有效性。

原文摘要 · Abstract (English)

In the instruction fine-tuning of large language models (LLMs), it is widely recognized that a few high-quality instructions are superior to a large number of low-quality instructions. At present, many instruction selection methods have been proposed, but most of these methods select instruction based on heuristic quality metrics, and only consider data selection before training. These designs lead to insufficient optimization of instruction fine-tuning, and fixed heuristic indicators are often difficult to optimize for specific tasks. Therefore, we design a dynamic, task-objective-driven instruction selection framework RAISE(Reinforced Adaptive Instruction SElection), which incorporates the entire instruction fine-tuning process into optimization, selecting instructions at each step based on the expected impact of each instruction on model performance improvement. Our approach is well interpretable and has strong task-specific optimization capabilities. By modeling dynamic instruction selection as a sequential decision-making process, we use RL to train our selection strategy. Extensive experiments and result analysis prove the superiority of our method compared with other instruction selection methods. Notably, RAISE achieves superior performance by updating only 1% of the training steps compared to full-data training, demonstrating its efficiency and effectiveness.

指令微调强化学习高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。