提出SINKFLEX-RL系统,提升长时程工具使用智能体的训练效率与内存可扩展性。
Efficient Reinforcement Learning for Long-Horizon Tool-Use Agentic Tasks

- 设计模块化训练系统,融合环境封装、数据流与自适应注意力机制。
- 在零售任务中验证奖励从0.25提升至0.44,轨迹奖励与训练得分同步上升。
- 优化注意力路径降低显存占用19.7%,支持8192 token长序列推理。
长时程工具使用智能体需对用户目标、领域策略、工具调用、模拟器状态及延迟可验证奖励进行推理。强化学习(RL)适合此场景,但多轮在线采样产生长上下文,且模型特定注意力层可能需定制掩码与学习型汇合归一化。本文提出SINKFLEX-RL,一个用于双控工具使用环境的模块化训练系统。该系统结合Gymnasium兼容环境包装器、类似VERL的采样数据流、无需独立价值模型的组相对策略优化,以及一种可保持因果与滑动窗口掩码下模型特有汇合缩放的汇合感知FlexAttention路径。在初步的Tau2Bench零售任务测试中,验证奖励(均值@1)从训练初期的0.25升至0.44,训练得分与轨迹奖励代理也持续上升。在固定配置内存基准测试中,优化后的注意力路径将4096令牌下的峰值显存从28.06GB降至22.52GB(降低19.7%),并在8192令牌配置下以25.53GB运行,而急切基线因显存不足无法完成。结果表明,集成环境接口、强化学习数据流与注意力核设计对实现内存可行的长时程智能体训练具有关键价值。
原文摘要 · Abstract (English)
Long-horizon tool-using agents must reason over user goals, domain policies, tool calls, simulator state, and delayed verifiable rewards. Reinforcement learning (RL) is a natural fit for this setting, but multi-turn on-policy rollouts create long contexts, while model-specific attention layers may require custom masks and learned sink normalization. We present SINKFLEX-RL, a modular training system for RL in dual-control tool-use environments. The system combines a Gymnasium-compatible environment wrapper, a VERL-style rollout dataflow, group-relative policy optimization without a separate value model, and a sink-aware FlexAttention path designed to preserve model-specific sink scaling under causal and sliding-window masks. In a preliminary Tau2Bench retail run, validation reward (mean@1) rises from 0.25 early in training to $0.44$ later in the observed training window, while training-score and trajectory-reward proxies also trend upward. In a fixed-configuration memory benchmark, the optimized attention path reduces peak VRAM from 28.06GB to 22.52GB at 4096 tokens, a $19.7\%$ reduction, and runs the measured 8192-token configuration using $25.53$~GB where the eager baseline runs out of memory. These results illustrate the value of integrating environment interfaces, RL dataflow, and attention-kernel design for memory-feasible long-horizon agent training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。