arXiv:2510.27630cs.AI2025-10被引 1

让人类只在关键时介入,低成本训练长任务AI代理。

Interaction as Intelligence Part II: Asynchronous Human-Agent Rollout for Long-Horizon Task Training

  • 人类仅在代理偏离正轨时干预,降低标注成本。
  • 训练后性能比无监督基线提升超50%,比无干预版本高28%。
  • 适合需长期规划的领域专用任务,如科研、编程等。

大型语言模型代理在自动编码、深度研究和图形界面操作等领域展现出强大潜力,但训练其完成需要数天甚至数月的长周期、领域专业化任务仍具挑战性。现有方法主要分为两类:一是依赖密集人工标注的行为克隆,成本过高;二是基于结果驱动的采样,常因有效正向轨迹稀少而失效。本文提出Apollo框架,融合异步人类指导与动作级数据过滤。该框架不要求人类全程跟随每一步,而是允许其在代理偏离有希望的轨迹时介入,提供先验知识或策略建议。这种轻量设计使持续交互超过30小时,显著降低数据收集成本。随后通过监督控制筛选劣质动作,防止错误传播。两者结合实现长周期环境下的可靠高效数据采集。在InnovatorBench上的实验表明,将GLM-4.5模型使用Apollo训练后,性能比未训练基线提升超50%,比无人类参与的变体提高28%。结果凸显了人机协同采样的关键作用,以及Apollo设计在处理长周期、领域专业化任务中的鲁棒性。

原文摘要 · Abstract (English)

Large Language Model (LLM) agents have recently shown strong potential in domains such as automated coding, deep research, and graphical user interface manipulation. However, training them to succeed on long-horizon, domain-specialized tasks remains challenging. Current methods primarily fall into two categories. The first relies on dense human annotations through behavior cloning, which is prohibitively expensive for long-horizon tasks that can take days or months. The second depends on outcome-driven sampling, which often collapses due to the rarity of valid positive trajectories on domain-specialized tasks. We introduce Apollo, a sampling framework that integrates asynchronous human guidance with action-level data filtering. Instead of requiring annotators to shadow every step, Apollo allows them to intervene only when the agent drifts from a promising trajectory, by providing prior knowledge, strategic advice, etc. This lightweight design makes it possible to sustain interactions for over 30 hours and produces valuable trajectories at a lower cost. Apollo then applies supervision control to filter out sub-optimal actions and prevent error propagation. Together, these components enable reliable and effective data collection in long-horizon environments. To demonstrate the effectiveness of Apollo, we evaluate it using InnovatorBench. Our experiments show that when applied to train the GLM-4.5 model on InnovatorBench, Apollo achieves more than a 50% improvement over the untrained baseline and a 28% improvement over a variant trained without human interaction. These results highlight the critical role of human-in-the-loop sampling and the robustness of Apollo's design in handling long-horizon, domain-specialized tasks.

人机协作长程任务智能代理数据采样

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。