解释大模型重复词语失败的根源，发现注意力陷阱是主因
Interpreting the Repeated Token Phenomenon in Large Language Models
- 发现注意力陷阱机制导致初始词被过度关注
- 长重复序列会破坏该神经回路，引发输出偏离
- 提出针对性修复方案，不损害模型整体性能
大语言模型在要求重复单个词时，常输出无关内容，这一未解缺陷构成安全漏洞。研究揭示该现象与‘注意力陷阱’有关——初始词获得异常高的注意力分数，是保障流畅性的关键机制。通过分析，识别出负责此现象的神经回路，并发现长重复序列会破坏该回路。该发现可推广至其他非重复序列的类似问题。为此提出一种靶向修复方法，有效解决重复失败问题，且不影响模型整体表现。本研究提供了一种可解释性驱动的诊断与修复路径，为构建更可靠、安全的模型提供洞见。
原文摘要 · Abstract (English)
Large Language Models (LLMs), despite their impressive capabilities, often fail to accurately repeat a single word when prompted to, and instead output unrelated text. This unexplained failure mode represents a vulnerability, allowing even end-users to diverge models away from their intended behavior. We aim to explain the causes for this phenomenon and link it to the concept of ``attention sinks'', an emergent LLM behavior crucial for fluency, in which the initial token receives disproportionately high attention scores. Our investigation identifies the neural circuit responsible for attention sinks and shows how long repetitions disrupt this circuit. We extend this finding to other non-repeating sequences that exhibit similar circuit disruptions. To address this, we propose a targeted patch that effectively resolves the issue without negatively impacting the model's overall performance. This study provides a mechanistic explanation for an LLM vulnerability, demonstrating how interpretability can diagnose and address issues, and offering insights that pave the way for more secure and reliable models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。