解释大模型理解上下文出错的原因，提出'令牌依赖冲突'新视角。
Racing Thoughts: Explaining Contextualization Errors in Large Language Models
- 用机制可解释性方法分析令牌间依赖关系，发现错误源于上下文整合顺序
- 实验证明在问号前未正确解析'bank'是导致误判的关键原因
- 适合关注大模型推理缺陷与可解释性的研究人员阅读
基于Transformer的语言模型之所以成功，很大程度上归功于其能从输入序列中整合相关上下文信息以生成回答或完成任务。然而，我们对模型实现这一能力所采用的算法知之甚少，对其失败模式也缺乏理解。例如，在提示「John is going fishing, so he walks over to the bank. Can he make an ATM transaction?」下，若模型未能将"bank"正确定义为地理特征而非金融机构，则可能错误回答"Yes"。本文提出大语言模型中的『竞争条件假说』（LLM Race Conditions Hypothesis），认为此类上下文误解源于令牌间的依赖关系（如"bank"必须在最终符号"?"整合信息前被正确理解）。通过多种机械可解释性技术，我们提供了相关性和因果性证据支持该假说，并建议在推理时进行干预以缓解此类错误。
原文摘要 · Abstract (English)
The profound success of transformer-based language models can largely be attributed to their ability to integrate relevant contextual information from an input sequence in order to generate a response or complete a task. However, we know very little about the algorithms that a model employs to implement this capability, nor do we understand their failure modes. For example, given the prompt "John is going fishing, so he walks over to the bank. Can he make an ATM transaction?", a model may incorrectly respond "Yes" if it has not properly contextualized "bank" as a geographical feature, rather than a financial institution. We propose the LLM Race Conditions Hypothesis as an explanation of contextualization errors of this form. This hypothesis identifies dependencies between tokens (e.g., "bank" must be properly contextualized before the final token, "?", integrates information from "bank"), and claims that contextualization errors are a result of violating these dependencies. Using a variety of techniques from mechanistic intepretability, we provide correlational and causal evidence in support of the hypothesis, and suggest inference-time interventions to address it.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。