揭示大模型处理生僻词的神经机制，发现其依赖分布式非模块化设计。
No Clustering, No Routing: How Transformers Actually Process Rare Tokens
- 通过神经元影响分析发现生僻词需额外专用神经元，形成双计算模式。
- 专用神经元在空间上分散分布，不构成模块化集群。
- 注意力机制不优先路由到专精神经元，体现训练驱动的柔性适应。
大语言模型在预测生僻词时表现不佳，但其内部专业化机制尚不明确。已有研究识别出遵循独特三阶段影响模式的专用「平台」神经元 \\cite{liu2025emergent}，但其功能组织仍未知。本文通过对 GPT-2 XL 与 Pythia 模型进行神经元影响分析、基于图的聚类及注意力头消融实验，发现：(1) 生僻词处理需额外的平台神经元，超出仅覆盖常见词的幂律调节范围，形成双计算模式；(2) 平台神经元在空间上分散分布，未形成模块化集群；(3) 注意力机制对专精神经元无偏好性路由。结果表明，生僻词专业化源于分布式、训练驱动的分化，而非架构上的模块化，既保持上下文敏感性，又实现自适应容量分配。
原文摘要 · Abstract (English)
Large language models struggle with rare token prediction, yet the mechanisms driving their specialization remain unclear. Prior work identified specialized ``plateau'' neurons for rare tokens following distinctive three-regime influence patterns \cite{liu2025emergent}, but their functional organization is unknown. We investigate this through neuron influence analyses, graph-based clustering, and attention head ablations in GPT-2 XL and Pythia models. Our findings show that: (1) rare token processing requires additional plateau neurons beyond the power-law regime sufficient for common tokens, forming dual computational regimes; (2) plateau neurons are spatially distributed rather than forming modular clusters; and (3) attention mechanisms exhibit no preferential routing to specialists. These results demonstrate that rare token specialization arises through distributed, training-driven differentiation rather than architectural modularity, preserving context-sensitive flexibility while achieving adaptive capacity allocation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。