无需额外计算，轻松看清大模型关注了哪些词。
VISTA: Visualization of Token Attribution via Efficient Analysis
- 通过删词测影响，从语义方向、强度和维度三方面评估词重要性。
- 不依赖反向传播，内存占用与原模型一致，适合部署时分析。
- 开源工具包支持复现，适合想理解AI决策过程的研究者和工程师。
理解大型语言模型如何处理提示信息仍是重大挑战。为揭示这一‘黑箱’，已有注意力可视化技术用于捕捉神经元级感知并解释模型对输入数据不同部分的关注方式。然而，多数现有方法针对特定模型架构（尤其是Transformer），通常需要反向传播，导致GPU内存使用量几乎翻倍且计算成本增加。目前缺乏轻量、模型无关的注意力可视化方法。本文提出一种模型无关的词元重要性可视化技术，以更好理解生成式AI系统如何感知并优先处理输入文本信息，且不增加额外计算开销。该方法结合扰动策略与三矩阵分析框架，生成反映词元层面贡献的相关性图谱。框架包含：(1) 角度偏移矩阵，捕捉语义方向变化；(2) 幅度偏移矩阵，衡量语义强度变化；(3) 维度重要性矩阵，评估各向量维度的贡献。通过系统移除每个词元并测量其在上述三个互补维度上的影响，推导出综合重要性得分，提供一种细致且数学严谨的词元显著性度量。为支持可复现性和促进广泛应用，我们公开所有提出的可解释性技术实现，代码与资源已发布于 https://github.com/Infosys/Infosys-Responsible-AI-Toolkit。
原文摘要 · Abstract (English)
Understanding how Large Language Models (LLMs) process information from prompts remains a significant challenge. To shed light on this "black box," attention visualization techniques have been developed to capture neuron-level perceptions and interpret how models focus on different parts of input data. However, many existing techniques are tailored to specific model architectures, particularly within the Transformer family, and often require backpropagation, resulting in nearly double the GPU memory usage and increased computational cost. A lightweight, model-agnostic approach for attention visualization remains lacking. In this paper, we introduce a model-agnostic token importance visualization technique to better understand how generative AI systems perceive and prioritize information from input text, without incurring additional computational cost. Our method leverages perturbation-based strategies combined with a three-matrix analytical framework to generate relevance maps that illustrate token-level contributions to model predictions. The framework comprises: (1) the Angular Deviation Matrix, which captures shifts in semantic direction; (2) the Magnitude Deviation Matrix, which measures changes in semantic intensity; and (3) the Dimensional Importance Matrix, which evaluates contributions across individual vector dimensions. By systematically removing each token and measuring the resulting impact across these three complementary dimensions, we derive a composite importance score that provides a nuanced and mathematically grounded measure of token significance. To support reproducibility and foster wider adoption, we provide open-source implementations of all proposed and utilized explainability techniques, with code and resources publicly available at https://github.com/Infosys/Infosys-Responsible-AI-Toolkit
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。