arXiv:2503.06202cs.AIcs.LG2025-03ICLR被引 10

提出新方法,用输入利用度替代互信息,更精准定位模型真正依赖的输入部分。

Breaking Free from MMI: A New Frontier in Rationalization by Probing Input Utilization

  • 通过权重矩阵方向匹配度判断输入是否被有效利用,替代传统互信息准则。
  • 在4个文本和1个图分类数据集上,比MMI及其改进版更准确识别关键输入片段。
  • 方法简单高效,媲美大型语言模型,适合需要可解释性的模型调试与优化场景。

从完整输入中提取关键推理依据是可解释性研究的核心问题。目前最常用的基准标准是最大互信息(MMI)准则。本文首次揭示MMI存在边际收益递减问题:一旦部分理由被识别,剩余部分对提升互信息贡献极小,难以继续定位。与侧重复现预测结果的MMI不同,本文转向识别网络实际利用的输入部分。核心思想是:神经网络的权重矩阵通常低秩,其列向量仅能覆盖高维空间中的有限方向。若输入被充分使用,其表示范数会较高;反之,若输入主要落在这些方向之外(正交),则表示范数趋近于零,表现为无法利用的噪声。基于此,本文提出以理由候选的范数作为新目标函数,替代MMI。在四个文本分类数据集和一个图分类数据集上,使用GRU、BERT和GCN三种架构进行实验,结果表明该方法显著优于MMI及其改进版本。此外,与代表性大模型llama-3.1-8b-instruct对比,本方法表现相当,甚至在某些情况下更优。

原文摘要 · Abstract (English)

Extracting a small subset of crucial rationales from the full input is a key problem in explainability research. The most widely used fundamental criterion for rationale extraction is the maximum mutual information (MMI) criterion. In this paper, we first demonstrate that MMI suffers from diminishing marginal returns. Once part of the rationale has been identified, finding the remaining portions contributes only marginally to increasing the mutual information, making it difficult to use MMI to locate the rest. In contrast to MMI that aims to reproduce the prediction, we seek to identify the parts of the input that the network can actually utilize. This is achieved by comparing how different rationale candidates match the capability space of the weight matrix. The weight matrix of a neural network is typically low-rank, meaning that the linear combinations of its column vectors can only cover part of the directions in a high-dimensional space (high-dimension: the dimensions of an input vector). If an input is fully utilized by the network, {it generally matches these directions (e.g., a portion of a hypersphere), resulting in a representation with a high norm. Conversely, if an input primarily falls outside (orthogonal to) these directions}, its representation norm will approach zero, behaving like noise that the network cannot effectively utilize. Building on this, we propose using the norms of rationale candidates as an alternative objective to MMI. Through experiments on four text classification datasets and one graph classification dataset using three network architectures (GRUs, BERT, and GCN), we show that our method outperforms MMI and its improved variants in identifying better rationales. We also compare our method with a representative LLM (llama-3.1-8b-instruct) and find that our simple method gets comparable results to it and can sometimes even outperform it.

可解释性神经网络推理依据低秩分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。