大模型自动生成的反事实解释既不简洁也不可靠，难以真实反映决策逻辑。
LLMs Don't Know Their Own Decision Boundaries: The Unreliability of Self-Generated Counterfactual Explanations
- 让大模型修改输入来生成反事实解释，但多数改动过大
- 要求简洁时又改得过少，无法改变预测结果
- 解释不可靠，不适合高风险场景使用
为与人类有效协作，语言模型需能以自然语言解释其决策。本文研究一种自生成反事实解释（SCEs）：模型通过修改输入使预测结果不同来说明自身判断。我们评估了这些解释是否有效（能达成预期结果）且最小化（改动最少）。实验发现，大模型生成的SCEs虽通常有效，但改动远超必要；当要求最小化时，它们常做过度微小的修改，无法改变预测。该有效性与最小性之间的权衡在多个大模型、数据集和评估设置中均一致。结果表明，SCEs要么无效，要么误导，不宜作为可信赖的可解释性工具。部署大模型于高风险场景时，必须考虑其不可靠自解释带来的下游决策风险。代码已公开于 https://github.com/HarryMayne/SCEs。
原文摘要 · Abstract (English)
To collaborate effectively with humans, language models must be able to explain their decisions in natural language. We study a specific type of self-explanation: self-generated counterfactual explanations (SCEs), where a model explains its prediction by modifying the input such that it would have predicted a different outcome. We evaluate whether LLMs can produce SCEs that are valid, achieving the intended outcome, and minimal, modifying the input no more than necessary. When asked to generate counterfactuals, we find that LLMs typically produce SCEs that are valid, but far from minimal, offering little insight into their decision-making behaviour. Worryingly, when asked to generate minimal counterfactuals, LLMs typically make excessively small edits that fail to change predictions. The observed validity-minimality trade-off is consistent across several LLMs, datasets, and evaluation settings. Our findings suggest that SCEs are, at best, an ineffective explainability tool and, at worst, can provide misleading insights into model behaviour. Proposals to deploy LLMs in high-stakes settings must consider the impact of unreliable self-explanations on downstream decision-making. Our code is available at https://github.com/HarryMayne/SCEs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。