arXiv:2605.09698cs.AI2026-05被引 1

测试数据科学代理在任务模糊时的误判问题,发现其常沉默提交错误方案。

Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agents

论文配图:Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agents
图 1 · 摘自论文原文
  • 构建两套诊断基准,分别检测目标与评估指标的模糊性
  • 51个目标模糊任务和61个指标模糊任务,均导致性能下降
  • 能提问可显著恢复性能,但模型无法判断何时该问

随着数据科学代理从协作者转向自主执行者,隐性任务误设成为关键故障模式。代理会默默采纳看似合理但实际错误的任务定义,生成看似正确且可运行的成果,掩盖了对任务不明确的误判。现有基准仅评估流程是否运行,忽略任务是否被正确理解。本文提出 Ambig-DS,包含两个诊断套件:基于 DSBench 的目标模糊套件(Ambig-DS-Target,51 个任务)和基于 MLE-bench 的评估指标模糊套件(Ambig-DS-Objective,61 个任务),保留原评价器。每个任务配对原始明确版本与经控制编辑产生的模糊变体,经人与大模型验证确认每种变体存在多个合理解释且影响决策。独立分析显示模糊性降低表现。在五种不同水平的代理中发现:(i) 失败为沉默承诺——目标模糊任务提交错误目标,指标模糊任务提交错误指标或无作为基线;(ii) 允许提一个澄清问题可在理想条件下大幅恢复性能,表明信息缺失是主要退化原因;(iii) 但代理无法可靠判断何时提问:宽松提示导致在清晰任务上过度提问,保守提示则导致在模糊任务上沉默默认。识别任务定义不完整,而非流程执行,才是当前评估中的缺失瓶颈。

原文摘要 · Abstract (English)

As data-science agents shift from co-pilots to auto-pilots, silent misframing becomes a critical failure mode. Agents quietly commit to plausible but unintended task framings, producing clean, executable artifacts that hide their incorrect assessment of the task. Existing benchmarks score whether the pipeline runs, ignoring whether the agent recognized the task was underspecified. We introduce Ambig-DS, two diagnostic suites: one for prediction-target ambiguity (Ambig-DS-Target, 51 tasks built on DSBench, a tabular modeling benchmark) and one for evaluation-objective ambiguity (Ambig-DS-Objective, 61 tasks built on MLE-bench, a Kaggle-style ML competition benchmark), constructed so that scoring uses each source benchmark's original evaluator. For every task we pair the original, fully specified version with an ambiguous variant produced by controlled edits; a human-and-LLM verification pipeline confirms each variant admits multiple plausible interpretations with decision-relevant consequences. The suites are analyzed independently and ambiguity lowers performance in both. Across five agents spanning efficient to frontier-class models, we find in our controlled diagnostic setting: (i) failures are silent commitments: wrong-target submissions on Target, wrong-metric or non-committal baseline submissions on Objective, rather than execution errors; (ii) allowing the agent to ask one clarifying question recovers much of the loss under idealized conditions, suggesting missing framing information drives a substantial part of the observed degradation; but (iii) agents cannot reliably tell when to use it: permissive prompts induce over-asking on clear tasks, while conservative prompts induce silent defaulting on ambiguous ones. Recognizing target and objective underspecification, not pipeline execution, is the bottleneck missing from standard DS-agent evaluations.

任务模糊智能代理评估基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。