揭示语言模型中心理语言学特征的层级分布规律。
Where meaning lives: Layer-wise accessibility of psycholinguistic features in encoder and decoder language models
- 通过分层探测58项心理语言学特征，对比不同嵌入方法
- 最终层表示通常非最优，词汇特征早现，体验与情感特征晚现
- 方法选择和模型架构共同决定意义在模型中的位置
理解变换器语言模型如何编码具有心理学意义的意义内容，对理论和实践均至关重要。我们对10个变换器模型（涵盖仅编码器与仅解码器架构）进行了系统性的分层探测研究，覆盖58项心理语言学特征，并比较了三种嵌入提取方法。结果显示，意义的表征定位高度依赖于方法：上下文嵌入相比孤立嵌入展现出更高的特征特异性与不同的层间分布模式。在所有模型和方法中，最终层表示很少是线性探测恢复心理语言学信息的最佳选择。尽管存在差异，各类模型仍表现出一致的语义维度深度排序规律：词汇属性在早期达到峰值，而体验性与情感性维度则在后期更显著。这些结果表明，语言模型中意义的‘栖息地’取决于方法选择与架构约束的相互作用。
原文摘要 · Abstract (English)
Understanding where transformer language models encode psychologically meaningful aspects of meaning is essential for both theory and practice. We conduct a systematic layer-wise probing study of 58 psycholinguistic features across 10 transformer models, spanning encoder-only and decoder-only architectures, and compare three embedding extraction methods. We find that apparent localization of meaning is strongly method-dependent: contextualized embeddings yield higher feature-specific selectivity and different layer-wise profiles than isolated embeddings. Across models and methods, final-layer representations are rarely optimal for recovering psycholinguistic information with linear probes. Despite these differences, models exhibit a shared depth ordering of meaning dimensions, with lexical properties peaking earlier and experiential and affective dimensions peaking later. Together, these results show that where meaning "lives" in transformer models reflects an interaction between methodological choices and architectural constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。