让失败回复变成功，用重播策略提升指令跟随的强化学习效率
Replay Failures as Successes: Sample-Efficient Reinforcement Learning for Instruction Following
- 将失败尝试中满足的约束重播为成功样本，实现样本高效学习
- 在多个任务上用更少计算量达到良好性能，仅需二元奖励信号
- 适合资源有限但需精准遵循复杂指令的场景
强化学习（RL）在对齐大语言模型（LLM）以遵循各种约束的指令方面展现出潜力。然而，RL改进依赖于高质量、成功的响应采样；初始模型常因能力不足无法生成满足所有约束的回复，导致奖励稀疏或难以区分，阻碍学习进程。本文提出一种新的样本高效强化学习框架——事后悔溯重播（HiR），采用先选择后重写策略，基于事后满足的约束将失败尝试重播为成功样本。在重播样本和原始样本上同时进行强化学习，理论化为指令与回复层面的双重偏好学习，仅需二元奖励信号即可实现高效优化。大量实验表明，该方法在多种指令跟随任务中表现优异，且计算开销更低。代码与数据集已开源。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) has shown promise for aligning Large Language Models (LLMs) to follow instructions with various constraints. Despite the encouraging results, RL improvement inevitably relies on sampling successful, high-quality responses; however, the initial model often struggles to generate responses that satisfy all constraints due to its limited capabilities, yielding sparse or indistinguishable rewards that impede learning. In this work, we propose Hindsight instruction Replay (HiR), a novel sample-efficient RL framework for complex instruction following tasks, which employs a select-then-rewrite strategy to replay failed attempts as successes based on the constraints that have been satisfied in hindsight. We perform RL on these replayed samples as well as the original ones, theoretically framing the objective as dual-preference learning at both the instruction- and response-level to enable efficient optimization using only a binary reward signal. Extensive experiments demonstrate that the proposed HiR yields promising results across different instruction following tasks, while requiring less computational budget. Our code and dataset is available at https://github.com/sastpg/HIR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。