发现重复神经元影响上下文学习效果，可调控输出重复性。
Understanding and Controlling Repetition Neurons and Induction Heads in In-Context Learning
- 从重复神经元视角分析模型对重复输入的响应机制。
- 深层重复神经元更显著影响上下文学习表现。
- 可减少重复输出，同时保持强上下文学习能力，适合优化生成质量。
本文研究大语言模型（LLMs）识别重复输入模式的能力与其在上下文学习（ICL）中的表现之间的关系。与以往主要关注注意力头的工作不同，本文从技能神经元的角度出发，特别关注重复神经元。实验表明，这些神经元对ICL性能的影响随其所在层深度而异。通过对比重复神经元与归纳头（induction heads）的作用，进一步提出可在保持强大ICL能力的同时降低重复输出的调控策略。
原文摘要 · Abstract (English)
This paper investigates the relationship between large language models' (LLMs) ability to recognize repetitive input patterns and their performance on in-context learning (ICL). In contrast to prior work that has primarily focused on attention heads, we examine this relationship from the perspective of skill neurons, specifically repetition neurons. Our experiments reveal that the impact of these neurons on ICL performance varies depending on the depth of the layer in which they reside. By comparing the effects of repetition neurons and induction heads, we further identify strategies for reducing repetitive outputs while maintaining strong ICL capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。