arXiv:2604.09408cs.AI2026-04被引 8

测试智能体何时该求助,发现主流模型判断力严重不足。

HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?

  • 设计新基准HiL-Bench,通过隐藏障碍逼真模拟求助决策场景。
  • 所有前沿模型在缺信息时表现暴跌,最高仅恢复原性能的30%。
  • 揭示三类求助缺陷,证明判断失误是模型共性短板而非任务特例。

前沿编程智能体在信息完整时能解决复杂任务,但面对不完整或模糊需求时会崩溃。问题不在能力,而在判断力:是否该自主执行或主动求助。现有基准忽略此缺陷,只提供明确指令并仅奖励执行正确性,导致错误猜测与主动求证得分相同。本文提出人机协同基准HiL-Bench,用于评估这一选择性升级能力。每个任务包含经人工验证的阻碍点(缺失信息、模糊请求、矛盾信息),需通过逐步探索才能暴露,而非直接可见。核心指标Ask-F1是提问精确率与阻碍召回率的调和平均值,结构上防止通过刷问句作弊。在SWE和文本转SQL领域评估显示存在普遍的判断差距:无一模型在信息缺失时恢复超过其全信息性能的30%。失败分析揭示三大求助模式:过度自信且未识别差距;高不确定性但持续出错;广泛粗略求助无自我修正。这些一致模式证实求助能力差是模型层面缺陷,非任务特异。基于塑造后的Ask-F1奖励进行强化学习训练表明,判断力可被训练提升:32B模型不仅改善求助质量,还提高任务通过率,且优势跨领域迁移。模型未学特定领域求助规则,而是学会识别不可解的不确定性并据此行动。

原文摘要 · Abstract (English)

Frontier coding agents solve complex tasks when given complete context but collapse when specifications are incomplete or ambiguous. The bottleneck is not raw capability, but judgment: knowing when to act autonomously and when to ask for help. Current benchmarks are blind to this failure mode. They supply unambiguous detailed instructions and solely reward execution correctness, so an agent that makes a lucky guess for a missing requirement will score identically to one that would have asked to be certain. We present HiL-Bench (Human-in-the-Loop Benchmark) to measure this selective escalation skill. Each task contains human-validated blockers (missing information, ambiguous requests, contradictory information) that surface only through progressive exploration, not upfront inspection. Our core metric, Ask-F1, the harmonic mean of question precision and blocker recall, captures the tension between over-asking and silent guessing; its structure architecturally prevents gaming through question spam. Evaluation across SWE and text-to-SQL domains reveals a large universal judgment gap: no frontier model recovers more than a fraction of its full-information performance when deciding whether to ask. Failure analysis identifies three key help-seeking patterns: overconfident wrong beliefs with no gap detection; high uncertainty detection yet persistent errors; broad, imprecise escalation without self-correction. These consistent patterns confirm poor help-seeking is a model-level flaw, not task-specific. RL training on shaped Ask-F1 reward shows judgment is trainable: a 32B model improves both help-seeking quality and task pass rate, with gains that transfer across domains. The model does not learn domain-specific heuristics for when to ask; it learns to detect unresolvable uncertainty and act on it.

智能体人机协作评估基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。