用最小对立对分析大模型的语义表征,揭示其语言知识本质。
Linguistic Minimal Pairs Elicit Linguistic Similarity in Large Language Models
- 通过最小对立对对比激活差异,量化语言相似性
- 高资源语言中模型间一致性更强,细粒度理论分类更匹配
- 适合语言学与AI交叉研究者,理解模型如何学习语言
本文提出一种新方法,利用语言学中的最小对立对来探测大型语言模型(LLMs)内部的语言表征。通过测量不同最小对立对在模型激活差异上的相似性,我们量化并深入理解了模型所捕捉的语言知识。大规模实验覆盖100多个LLM和150,000个最小对立对,涵盖三种语言,从四个关键维度揭示语言相似性的特征:跨模型的一致性、与理论分类的关系、对语义上下文的依赖性,以及跨语言对齐情况。结果表明:1)语言相似性受训练数据影响显著,高资源语言中模型间一致性更高;2)语言相似性与精细理论分类高度一致,但与宽泛分类关联较弱;3)语言相似性与语义相似性相关性弱,具有明显上下文依赖性;4)模型在跨语言语言现象理解上对齐程度有限。本工作展示了最小对立对作为窥探大模型神经表征的窗口的潜力,深化了模型与语言学理论之间的联系。代码与数据已开源。
原文摘要 · Abstract (English)
We introduce a novel analysis that leverages linguistic minimal pairs to probe the internal linguistic representations of Large Language Models (LLMs). By measuring the similarity between LLM activation differences across minimal pairs, we quantify the and gain insight into the linguistic knowledge captured by LLMs. Our large-scale experiments, spanning 100+ LLMs and 150k minimal pairs in three languages, reveal properties of linguistic similarity from four key aspects: consistency across LLMs, relation to theoretical categorizations, dependency to semantic context, and cross-lingual alignment of relevant phenomena. Our findings suggest that 1) linguistic similarity is significantly influenced by training data exposure, leading to higher cross-LLM agreement in higher-resource languages. 2) Linguistic similarity strongly aligns with fine-grained theoretical linguistic categories but weakly with broader ones. 3) Linguistic similarity shows a weak correlation with semantic similarity, showing its context-dependent nature. 4) LLMs exhibit limited cross-lingual alignment in their understanding of relevant linguistic phenomena. This work demonstrates the potential of minimal pairs as a window into the neural representations of language in LLMs, shedding light on the relationship between LLMs and linguistic theory. Codes and data are available at https://github.com/ChenDelong1999/Linguistic-Similarity
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。