解释模型不可靠，科学发现需警惕假象。
Reliability, Faithfulness, and the Limits of Post-hoc Explanations of Opaque Scientific Models
- 用事后解释推断现象机制存在逻辑漏洞
- 模型可靠且解释忠实仍不足以为结构定论
- 适合关注模型可信度与科学推理的读者
事后解释方法常被用于解读科学机器学习模型，其成果被视为对模型训练现象的洞察。通常认为，只要模型可靠且解释忠实，这一推断就成立。但本文指出这并不充分：可靠性仅验证预测与现象结果一致，忠实地仅确保解释与模型匹配，却无法验证模型是否真实反映现象的内在机制——而这正是结构性主张所必需的。即便在外部证据支持下，该推导链也仅能生成候选假设，无法独立支撑关于现象真实结构的结论。
原文摘要 · Abstract (English)
Post-hoc explanation methods are routinely used to interpret scientific machine learning models, with the deliverable understood to be insight into the phenomenon the model has been trained on. The transition may be taken to be secured once the model is reliable enough and the explanation faithful enough. We argue it is not. Reliability checks that the model's predictions match the phenomenon's outcomes, and faithfulness checks that the explanation matches the model, but neither checks whether the model works as the phenomenon works, which is what a claim about structure requires. The chain can support candidate hypotheses under external corroboration, but it cannot, on its own, support claims about how the phenomenon is in fact structured.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。