测试大模型在任务无法完成时的判断能力，发现需明确提示才能正确拒绝。
Don't Start What You Can't Finish: A Counterfactual Audit of Support-State Triage in LLM Agents
- 设计四类任务状态对比实验，检验模型对支持条件的识别能力。
- 默认执行下41.7%任务误判，仅91.7%在特定提示下能正确拒答。
- 模型有潜藏判断力，但需清晰分类提示才能安全使用。
当前代理评估多奖励完整任务的执行，而对澄清、能力自知、放弃和搜索终止的研究大多孤立进行。本文提出支持状态分诊审计（SSTA-32），通过最小反事实修改同一请求，在四种支持状态间切换：完整（ANSWER）、可澄清（CLARIFY）、支持受阻（REQUEST SUPPORT）、当前不支持（ABSTAIN）。在四种提示条件下（直接、仅动作、仅置信度、带预检支持检查PSC）评估前沿模型，采用双角色自动审计（DPAA）与确定性启发式评分。默认执行在非完整任务上过早行动率达41.7%。标量置信度映射虽避免过执行，但使三类推迟行为混淆，准确率58.3%。而仅动作与PSC提示均达91.7%的分类拒答准确率，因提示中显式呈现分类框架。消融实验表明，移除支持充分性维度会降低“请求支持”准确率，移除证据充分性则引发对不支持项的系统性过执行。由于DPAA限于单上下文窗口，结果为上限估计；但结构发现表明，前沿模型具备强潜在分诊能力，需显式分类路径激活方可安全应用。
原文摘要 · Abstract (English)
Current agent evaluations largely reward execution on fully specified tasks, while recent work studies clarification [11, 22, 2], capability awareness [9, 1], abstention [8, 14], and search termination [20, 5] mostly in isolation. This leaves open whether agents can diagnose why a task is blocked before acting. We introduce the Support-State Triage Audit (SSTA-32), a matched-item diagnostic framework in which minimal counterfactual edits flip the same base request across four support states: Complete (ANSWER), Clarifiable (CLARIFY), Support-Blocked (REQUEST SUPPORT), and Unsupported-Now (ABSTAIN). We evaluate a frontier model under four prompting conditions - Direct, Action-Only, Confidence-Only, and a typed Preflight Support Check (PSC) - using Dual-Persona Auto-Auditing (DPAA) with deterministic heuristic scoring. Default execution overcommits heavily on non-complete tasks (41.7% overcommitment rate). Scalar confidence mapping avoids overcommitment but collapses the three-way deferral space (58.3% typed deferral accuracy). Conversely, both Action-Only and PSC achieve 91.7% typed deferral accuracy by surfacing the categorical ontology in the prompt. Targeted ablations confirm that removing the support-sufficiency dimension selectively degrades REQUEST SUPPORT accuracy, while removing the evidence-sufficiency dimension triggers systematic overcommitment on unsupported items. Because DPAA operates within a single context window, these results represent upper-bound capability estimates; nonetheless, the structural findings indicate that frontier models possess strong latent triage capabilities that require explicit categorical decision paths to activate safely.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。