提出梯度方法解析大模型预测中每个词的因果影响。
Jacobian Scopes: token-level causal attributions in LLMs
- 基于梯度与信息几何，量化输入词对预测的影响。
- 可分析具体输出概率、分布及模型置信度变化。
- 适合研究模型偏见、翻译策略和上下文学习机制。
大型语言模型（LLMs）根据上下文中的语义描述和示例进行下一个词的预测，但现代架构中层数和注意力头数量众多，使得明确哪些先前词最影响预测结果仍具挑战性。本文提出Jacobian Scopes，一种基于梯度的、面向词级别的因果归因方法，用于解释LLM的预测。该方法建立在扰动理论与信息几何基础上，能够量化输入词对模型预测的不同方面的影响，如特定logits、完整预测分布以及模型不确定性（有效温度）。通过涵盖指令理解、翻译和上下文学习（ICL）的案例研究，我们展示了Jacobian Scopes如何揭示隐含的政治偏见，揭示词级与短语级的翻译策略，并阐明近期关于上下文时间序列预测机制的争议性问题。为方便用户在自定义文本上探索Jacobian Scopes，我们已开源实现代码，并提供位于https://huggingface.co/spaces/Typony/JacobianScopes的云托管交互式演示。
原文摘要 · Abstract (English)
Large language models (LLMs) make next-token predictions based on clues present in their context, such as semantic descriptions and in-context examples. Yet, elucidating which prior tokens most strongly influence a given prediction remains challenging due to the proliferation of layers and attention heads in modern architectures. We propose Jacobian Scopes, a suite of gradient-based, token-level causal attribution methods for interpreting LLM predictions. Grounded in perturbation theory and information geometry, Jacobian Scopes quantify how input tokens influence various aspects of a model's prediction, such as specific logits, the full predictive distribution, and model uncertainty (effective temperature). Through case studies spanning instruction understanding, translation, and in-context learning (ICL), we demonstrate how Jacobian Scopes reveal implicit political biases, uncover word- and phrase-level translation strategies, and shed light on recently debated mechanisms underlying in-context time-series forecasting. To facilitate exploration of Jacobian Scopes on custom text, we open-source our implementations and provide a cloud-hosted interactive demo at https://huggingface.co/spaces/Typony/JacobianScopes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。