arXiv:2605.15565cs.LGcs.AI2026-05

AstraFlow让大模型智能体强化学习更高效,支持多策略协同与跨区域计算。

AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMs

论文配图:AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMs
图 1 · 摘自论文原文
  • 采用数据流架构解耦推理、数据管理和训练组件,实现自主运行
  • 多策略协作训练效率提升2.7倍,精度相当或更优
  • 无需改代码即可支持弹性扩展和异构资源调度,适合复杂智能体系统

强化学习(RL)正被广泛用于提升大语言模型的推理、编码与工具使用能力,但智能体强化学习仍成本高昂。将RL扩展至智能体大模型需支持复杂工作负载,包括多策略协同训练,并高效利用弹性、异构及跨区域计算资源。现有LLM RL系统仅部分支持这些能力,且每次功能扩展常需额外系统工程。这源于以训练器为中心的控制架构和缺乏对RL组件的系统性抽象。为此,我们提出AstraFlow——一种面向数据流的强化学习系统,用可编程组件抽象取代传统以训练器为中心的控制方式。在AstraFlow中,回放服务、数据流管理与训练被解耦为独立自治组件,使系统原生支持复杂的多策略智能体RL任务,并高效利用多样化算力资源。我们在数学、代码、搜索及AgentBench任务上评估了AstraFlow,结果表明,同一系统无需修改底层代码即可支持多策略训练、弹性伸缩、异构跨区域执行和可组合的数据算法。在多策略协同训练中,其训练速度比现有系统快2.7倍,精度相当或更优。

原文摘要 · Abstract (English)

Reinforcement learning (RL) is increasingly used to improve the reasoning, coding, and tool-use capabilities of large language models, but agentic RL remains prohibitively expensive. Scaling RL to agentic LLMs requires supporting complex workloads, including multi-policy collaborative training, while efficiently using elastic, heterogeneous, and cross-region compute resources. Existing LLM RL systems support some of these capabilities, but each new extension often requires dedicated system engineering. This burden arises from trainer-centered control architectures and the lack of principled abstractions for RL system components. To address these limitations, we propose AstraFlow, a dataflow-oriented RL system that replaces conventional trainer-centered control with principled component abstractions. In AstraFlow, rollout services, dataflow management, and training are decoupled into autonomous components, enabling the system to natively support complex multi-policy agentic RL workloads and efficiently exploit diverse compute resources. We evaluate AstraFlow across math, code, search, and AgentBench workloads, showing that the same system supports multi-policy training, elastic scaling, heterogeneous cross-region execution, and composable data algorithms without system-level code changes. In multi-policy collaborative training, AstraFlow achieves comparable or better accuracy than existing RL systems while speeding up training time by 2.7x.

强化学习大模型智能体系统优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。