探究大模型如何表征动词短语的内部表示,发现中层最擅长区分类型。
Probing Internal Representations of Multi-Word Verbs in Large Language Models
- 用探测分类器分析BERT各层对动词短语的表征,定位关键识别层。
- 中层分类准确率最高,但两类动词在向量空间中线性不可分。
- 揭示语言结构与神经网络非线性表征的深层关联,适合语言学与模型解释研究者。
本研究考察了基于Transformer的大语言模型(LLM)中动词+介词/副词组合(即多词动词)的内部表征,特别关注这些模型在不同神经网络层中捕捉词汇和句法属性的能力。以BERT架构为例,分析其对两类动词短语——如'give up'的短语动词和'look at'的介词动词——的表征。方法包括在内部表示上训练探测分类器,分别在词级和句级进行分类。结果表明,模型中层达到最高分类准确率。为进一步分析区分特征的本质,采用广义判别值(GDV)进行数据可分性测试,结果显示两类动词间线性可分性较弱,但探测分类器仍能实现高准确率,表明这些语言类别的表征可能是非线性可分的。这与以往研究一致:神经网络中的语言差异并不总以线性方式编码。研究从计算角度支持了基于使用的动词短语表征理论,并突显了神经网络架构与语言结构之间的复杂互动。
原文摘要 · Abstract (English)
This study investigates the internal representations of verb-particle combinations, called multi-word verbs, within transformer-based large language models (LLMs), specifically examining how these models capture lexical and syntactic properties at different neural network layers. Using the BERT architecture, we analyze the representations of its layers for two different verb-particle constructions: phrasal verbs like 'give up' and prepositional verbs like 'look at'. Our methodology includes training probing classifiers on the internal representations to classify these categories at both word and sentence levels. The results indicate that the model's middle layers achieve the highest classification accuracies. To further analyze the nature of these distinctions, we conduct a data separability test using the Generalized Discrimination Value (GDV). While GDV results show weak linear separability between the two verb types, probing classifiers still achieve high accuracy, suggesting that representations of these linguistic categories may be non-linearly separable. This aligns with previous research indicating that linguistic distinctions in neural networks are not always encoded in a linearly separable manner. These findings computationally support usage-based claims on the representation of verb-particle constructions and highlight the complex interaction between neural network architectures and linguistic structures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。