让大模型强化学习更快更稳，异步训练也能高效用。
Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training
- 用轨迹平衡思想处理异步数据，解耦探索与学习过程。
- 在数学、偏好对齐等任务上提速4倍以上，性能优于在线DPO。
- 适合大规模并行训练，特别适合数据生成速度高的场景。
强化学习是大语言模型后训练的关键环节。然而，传统在线策略算法对经验回放缓冲区中多样化的数据不鲁棒，而异步离线智能体可高效并行填充此类数据。我们提出轨迹平衡与异步（TBA）方法，通过严谨的离线策略目标实现高效学习。在数学、偏好调优及自动化红队测试任务中，我们对从Pythia 410M到Qwen 2.5 7B的模型进行后训练，结果显示TBA相比在线DPO和Dr. GRPO等强基线，在速度和性能上均有提升。即使异步程度增加，TBA仍保持高精度；同时带来4倍及以上加速。此外，奖励与近期性优先采样机制在数据生成规模扩大时进一步提升效果。代码已开源。
原文摘要 · Abstract (English)
Reinforcement learning (RL) is a critical component of large language model (LLM) post-training. However, on-policy algorithms used for post-training are not naturally robust to a diversified content of experience replay buffers, which asynchronous off-policy actors can efficiently populate in parallel to training. We propose efficiently learning on such off-policy data via Trajectory Balance with Asynchrony (TBA), an approach to asynchronous RL for LLMs that leverages the principled off-policy TB objective. On math, preference-tuning, and automated red-teaming tasks, we post-train models ranging from Pythia 410M to Qwen 2.5 7B, finding TBA offers speed and performance boosts over strong baselines like Online DPO and Dr. GRPO. Beyond TBA's performance benefits (high accuracy even as asynchrony grows) and speedups ($4\times$ or more), we show its reward- and recency-prioritizing sampling enable further gains as data generation is scaled. Our code is available at https://github.com/bbartoldson/TBA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。