发现语言模型中两种独立的复制机制：词级与字级。
The Dual-Route Model of Induction
- 区分词级与字级归纳头，分别处理语义和字面复制。
- 词级头能实现跨语言翻译，字级头专攻无意义文本复现。
- 词级头输出具语言无关性，暗示模型存储抽象词义。
以往研究揭示了上下文复制中的归纳头，其关注并促进单个标记的复制。本文发现一种新型归纳头：概念级归纳头，可复制完整词汇单位而非单个标记。这类头在训练过程中持续关注多标记词汇末端，与字级归纳头并行工作以复制有意义文本。我们证明，概念级归纳头负责语义任务（如词级翻译），而字级归纳头对仅能逐字完成的任务（如复制无意义标记）至关重要。两条路径独立运作：删除字级归纳头后，模型会将原样复制转为改写。通过修补概念级归纳头输出，发现其包含语言无关的词表示，可中介自然语言翻译，表明大语言模型能独立于语言或形式表征抽象词义。
原文摘要 · Abstract (English)
Prior work on in-context copying has shown the existence of induction heads, which attend to and promote individual tokens during copying. In this work we discover a new type of induction head: concept-level induction heads, which copy entire lexical units instead of individual tokens. Concept induction heads learn to attend to the ends of multi-token words throughout training, working in parallel with token-level induction heads to copy meaningful text. We show that these heads are responsible for semantic tasks like word-level translation, whereas token induction heads are vital for tasks that can only be done verbatim (like copying nonsense tokens). These two "routes" operate independently: we show that ablation of token induction heads causes models to paraphrase where they would otherwise copy verbatim. By patching concept induction head outputs, we find that they contain language-independent word representations that mediate natural language translation, suggesting that LLMs represent abstract word meanings independent of language or form.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。