LLM能学习到有语义梯度的句式表示，越典型句式在空间中越分离。
Meaning-infused grammar: Gradient Acceptability Shapes the Geometric Representations of Constructions in LLMs
- 用人类评分的偏好强度梯度分析句式表征
- 典型句式在激活空间中更分离，差异与语义功能相关
- 支持语言学中的构式理论，适合语言模型研究者
基于使用构式的语言观认为，语言由大量习得的形式-意义配对（构式）构成，其使用受语义或功能影响，具有等级性和概率性。本研究探究大语言模型内部表征是否反映这种功能驱动的梯度特性。通过分析Pythia-1.4B模型在5000对句子上的表征，这些句子系统性地改变为人类评定的双宾语（DO）或介词宾语（PO）偏好强度。几何分析显示，两种构式表征的可分性（以能量距离或Jensen-Shannon散度衡量）随偏好强度梯度系统性变化：越典型的构式样本在激活空间中占据更独立区域，而模糊句则可能属于任一构式。结果表明LLM学习了丰富、意义嵌入且分级的构式表征，并验证了几何度量在表征分析中的有效性。
原文摘要 · Abstract (English)
The usage-based constructionist (UCx) approach to language posits that language comprises a network of learned form-meaning pairings (constructions) whose use is largely determined by their meanings or functions, requiring them to be graded and probabilistic. This study investigates whether the internal representations in Large Language Models (LLMs) reflect the proposed function-infused gradience. We analyze representations of the English Double Object (DO) and Prepositional Object (PO) constructions in Pythia-$1.4$B, using a dataset of $5000$ sentence pairs systematically varied by human-rated preference strength for DO or PO. Geometric analyses show that the separability between the two constructions' representations, as measured by energy distance or Jensen-Shannon divergence, is systematically modulated by gradient preference strength, which depends on lexical and functional properties of sentences. That is, more prototypical exemplars of each construction occupy more distinct regions in activation space, compared to sentences that could have equally well have occured in either construction. These results provide evidence that LLMs learn rich, meaning-infused, graded representations of constructions and offer support for geometric measures for representations in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。