构建持续协作的智能中介,解决人机长期任务中的注意力错配问题。
JarvisBench: Always-on Intelligence Between Humans and Agents

- 设计双向协调机制,动态分配人类注意力给多个运行中的智能体。
- 在45个真实任务中验证,能准确识别需人决策的时刻并及时请求介入。
- 支持任意智能体运行时集成,适合研究人机协同与智能代理系统。
长周期智能体可持续执行,但人类注意力间歇且稀缺,导致双向协调难题:用户可能随时需要介入,而智能体在无人监控时遇到关键决策需人类判断。为此提出始终在线的注意力协调层——Jarvis,作为人与智能体之间的中介,合理分配人类注意力。引入 JarvisBench 评估这一协调机制的双向能力:一是用户提问时能否快速准确回答正在进行的任务状态;二是能否识别出需人类判断的时刻,适时发起请求并反馈结果以优化任务表现。评测包含45个智能体任务实例,涵盖20个单智能体任务和25个分属10个跨智能体项目的流水线任务,覆盖19个领域,均从2000多个候选任务中筛选改编而来。关键在于,人类介入需求在执行过程中自然产生,而非初始提示遗漏所致。JarvisBench 可无缝集成至任意智能体运行环境,无需修改其执行逻辑。参考实现还提供全双工语音接口,支持用户自然接入,实现实时注意力协调,助力后台智能体持续工作。通过分离执行与注意力协调,该基准为持续演进的智能体能力提供了稳定评估目标。
原文摘要 · Abstract (English)
Long-horizon agents can execute continuously, but human attention remains intermittent and scarce. This creates a bidirectional coordination problem: users may need immediate access to an agent while work continues in the background, whereas agents may encounter consequential decisions that require user judgment after the user has stopped monitoring execution. We posit an always-on attention-coordination layer---\textit{Jarvis}\footnote{Named after the fictional AI assistant in \textit{Iron Man}.}---that mediates this interface and allocates human attention across one or more working agents. We introduce \textit{JarvisBench} to evaluate both directions of this coordination: whether an intermediary can accurately and promptly answer user-initiated questions about ongoing work, and whether it can recognize when an agent requires user judgment, solicit that judgment at the right moment, and route it back to improve task outcomes. JarvisBench contains 45 agentic task instances: 20 single-agent tasks and 25 workstreams organized into 10 multi-agent projects. The tasks span 19 domains and were selected and adapted from more than 2,000 public candidates. Crucially, the need for user attention arises naturally during execution rather than from an obvious omission in the initial prompt. JarvisBench is designed to integrate with arbitrary agent runtimes without modifying their underlying execution loops. Our reference implementation further provides a full-duplex speech interface, allowing users to reach Jarvis naturally while timely attention coordination supports agents working in the background. By separating agent execution from attention coordination, JarvisBench provides a stable evaluation target as agent capabilities continue to improve.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。