检验大模型解释是否真实反映决策依据,发现多数解释不可靠。
Necessary or Sufficient? Evaluating LLM Explanations With Behavioural Evidence

- 用可控黑盒实验测试解释因子的必要性与充分性
- 平均相关性仅0.35~0.58,说明解释与实际影响不一致
- 超半数未被提及的因素影响力反而超过被列因素
大模型在代理工作流中常输出带解释的判断,如推荐顾问或评估提示风险。这些解释假设其提及的因素与实际决策行为一致。本文检验两种解释含义:必要性(改变因素则输出变化)与充分性(保留因素可维持输出)。通过两个合成场景——顾问推荐与风险判断——对八款来自Claude、GPT和Gemini系列的模型进行评估。采用黑盒干预测量每个因子的必要性与充分性得分。结果显示,在顾问推荐任务中,引用排名与必要性、充分性的斯皮尔曼相关系数分别为0.349和0.354;在风险判断任务中分别为0.431和0.580。此外,未被引用的因素在必要性下有57.6%的概率超过最低引用因子,在充分性下为58.1%;对应风险判断任务中分别为25.8%和8.9%。可见,被列出的前三个因素虽含信息,但并未可靠代表实际最具影响力的三个因素。该框架为代理监督中的解释提供黑盒可靠性验证,且仅针对单个模型决策。
原文摘要 · Abstract (English)
LLM decision components that can operate within agent workflows often produce action-relevant recommendations or judgements together with explanations. Operators may use the named factors to monitor a system, diagnose errors, or decide when to escalate an output. Such use assumes that the explanations agree with the component's observable decision behaviour. We test two interpretations of the named factors: necessity, meaning that changing a factor would change the output, and sufficiency, meaning that retaining it while removing other changeable information would preserve the output. We evaluate these interpretations in two synthetic use cases: recommending advisors to clients and judging prompts for harmfulness or risk. Models return an output and the top three factors that most influenced it. Controlled black-box interventions estimate a necessity score for each factor by measuring how often changing it changes the output, and a sufficiency score by measuring how often retaining it preserves the output. Across eight models from the Claude, GPT, and Gemini families, the mean Spearman correlations between the cited ranking and the necessity and sufficiency scores are 0.349 and 0.354 for advisor recommendation, and 0.431 and 0.580 for prompt monitoring. Furthermore, an uncited factor scores above the lowest-scoring cited factor in 57.6% of advisor responses under necessity and 58.1% under sufficiency; the corresponding prompt-monitoring rates are 25.8% and 8.9%. The cited top three contain useful information but do not reliably identify the three factors with the strongest measured influence under necessity or sufficiency. The framework provides a black-box reliability check for explanations used in agent oversight while remaining scoped to individual LLM decisions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。