解析大模型中看似无用特征的生成机制，揭示其与推理能力的关系。
Understanding the Emergence of Seemingly Useless Features in Next-Token Predictors
- 通过梯度信号分析，定位导致冗余特征出现的关键训练成分。
- 发现高/低影响特征多关联代码等形式化推理任务，非冗余。
- 适用于理解预训练大模型内部表征演化，适合模型可解释性研究者。
训练后的Transformer会计算出对预测下一个词看似冗余的抽象特征。我们识别了下一词预测目标中哪些梯度成分引发此现象，并提出方法估算这些成分对特定特征生成的影响。在小规模任务上验证后，应用于OthelloGPT的世界模型和小型语言模型的句法特征，揭示其起源。进一步应用于预训练大模型，发现对后续词影响极大或极小的特征，多与代码等形式化推理领域相关。本工作从训练过程视角，推进了对Transformer隐含特征的理解。
原文摘要 · Abstract (English)
Trained Transformers have been shown to compute abstract features that appear redundant for predicting the immediate next token. We identify which components of the gradient signal from the next-token prediction objective give rise to this phenomenon, and we propose a method to estimate the influence of those components on the emergence of specific features. After validating our approach on toy tasks, we use it to interpret the origins of the world model in OthelloGPT and syntactic features in a small language model. Finally, we apply our framework to a pretrained LLM, showing that features with extremely high or low influence on future tokens tend to be related to formal reasoning domains such as code. Overall, our work takes a step toward understanding hidden features of Transformers through the lens of their development during training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。