用可解释的检索机制提升文本预测准确率，接近黑箱模型性能。
Interpretable Next-token Prediction via the Generalized Induction Head
- 基于精确匹配与神经相似度的模糊匹配，构建可解释的文本预测模块。
- 在语言建模中比可解释基线提升25个百分点，接近大模型表现。
- 适用于需要理解推理过程的高风险场景，如脑科学与语言研究。
尽管大型Transformer模型在预测任务中表现优异，但其缺乏可解释性限制了其在高风险领域的应用。为此，我们提出广义归纳头模型（GIM），一种受大语言模型中“归纳头”现象启发的可解释文本预测模型。GIM是一个基于检索的模块，通过结合精确n-gram匹配与基于神经相似度的模糊匹配，在输入上下文中识别相似序列。我们在两种场景下评估GIM：语言建模和fMRI神经响应预测。在语言建模任务中，GIM相比可解释基线提升高达25个百分点，显著缩小了与黑箱大模型之间的差距；在fMRI任务中，神经响应预测性能提升20%，并揭示了大脑对语言的特异性。GIM为跨领域实现可解释性与高性能的统一提供了重要进展。代码已开源：https://github.com/ejkim47/generalized-induction-head。
原文摘要 · Abstract (English)
While large transformer models excel in predictive performance, their lack of interpretability restricts their usefulness in high-stakes domains. To remedy this, we propose the Generalized Induction-Head Model (GIM), an interpretable model for next-token prediction inspired by the observation of "induction heads" in LLMs. GIM is a retrieval-based module that identifies similar sequences in the input context by combining exact n-gram matching and fuzzy matching based on a neural similarity metric. We evaluate GIM in two settings: language modeling and fMRI response prediction. In language modeling, GIM improves next-token prediction by up to 25%p over interpretable baselines, significantly narrowing the gap with black-box LLMs. In an fMRI setting, GIM improves neural response prediction by 20% and offers insights into the language selectivity of the brain. GIM represents a significant step toward uniting interpretability and performance across domains. The code is available at https://github.com/ejkim47/generalized-induction-head.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。