检验大模型解释是否真能说明其推理过程,发现多数解释都不够充分。
What LLMs explain is not what they believe: Evaluating explanation sufficiency under models' own input beliefs

- 用模型自身生成基于解释的替代输入,捕捉其对输入的信念。
- 提出SCSuff度量,无需预设偏见即可评估解释充分性。
- 解释充分性随输入分布变化,且与模型大小无关,可从内部状态预测。
大语言模型(LLMs)在高风险领域部署时,常通过链式思考或事后理由等自由文本解释来证明输出合理性。然而,这些解释是否足够充分——即是否包含足以解释模型输出生成过程的信息——仍不明确。本文将经典的特征重要性充分性推广至任意解释,并证明解释充分性依赖于输入分布,必须明确定义。我们提出利用模型自身生成基于解释的替代输入,以捕捉其对可能输入的信念。定义自洽充分性为目标,并引入信息论度量SCSuff,实现无需预设偏见或捷径的自由文本解释评估。实验表明,SCSuff与目标扰动测试结果一致,且解释充分性随输入分布变化。研究发现,大多数模型解释不充分,且与模型规模、准确率或输出熵弱相关。最终标记隐藏状态分析显示,高低SCSuff得分可由内部表示预测,表明该度量可用于识别和改进充分解释。代码已开源:https://github.com/rajesh-lab/self-consistent-sufficiency。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly deployed in high-stakes domains, where free-text explanations such as chain-of-thought and post-hoc rationales are used to justify model outputs. Yet it remains unclear whether these explanations are sufficient, i.e., if they contain enough information to explain the model's output-generating process. We generalize classical sufficiency from feature attributions to arbitrary explanations and prove that explanation sufficiency can change depending on the input distribution, which must be explicitly defined for LLM explanations. We propose using the LLM itself to generate alternative inputs conditioned on an explanation, capturing its beliefs about possible inputs. We formalize self-consistent sufficiency as a goal for free-text explanations and introduce an information-theoretic metric, SCSuff, that enables evaluation of free-text explanations without relying on predefined biases or shortcuts. Our experiments show that SCSuff agrees with targeted perturbation tests where applicable and demonstrate that explanation sufficiency can vary with the input distribution. We find LLM explanations are generally insufficient and weakly correlated with model size, accuracy, or output entropy. Analysis of final-token hidden states shows that top and bottom SCSuff scores can be predicted from internal representations, suggesting that SCSuff can guide detection and improvement of sufficient LLM explanations. The code for this paper is available at https://github.com/rajesh-lab/self-consistent-sufficiency .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。