分析Transformer如何表示复合词，发现多数模型具强组合性。
How do Transformer Embeddings Represent Compositions? A Functional Analysis
- 用六种组合模型检验嵌入表示，线性回归最佳
- 向量加法表现接近最优，多数模型组合性强
- 适合研究语言表示与模型推理机制的读者
组合性是人类智能的关键，对推理和泛化至关重要。尽管基于Transformer的模型已成为语言建模的标准，但其对复合词的表示方式及其是否具备组合性仍不明确。本研究测试了Mistral、OpenAI Large、Google嵌入模型及BERT在组合性方面的表现。通过六种组合模型（加法、乘法、缩放、回归等）评估表示能力，发现岭回归虽为线性模型，却最能解释组合性；令人惊讶的是，经典向量加法模型表现几乎与其它模型相当。进一步验证表明，多数嵌入模型具有高度组合性，而BERT则显著较差。研究使用包含完全透明形容词-名词组合的合成数据集验证并可视化结果。整体上，本文对组合性进行了全面探究。
原文摘要 · Abstract (English)
Compositionality is a key aspect of human intelligence, essential for reasoning and generalization. While transformer-based models have become the de facto standard for many language modeling tasks, little is known about how they represent compound words, and whether these representations are compositional. In this study, we test compositionality in Mistral, OpenAI Large, and Google embedding models, and compare them with BERT. First, we evaluate compositionality in the representations by examining six diverse models of compositionality (addition, multiplication, dilation, regression, etc.). We find that ridge regression, albeit linear, best accounts for compositionality. Surprisingly, we find that the classic vector addition model performs almost as well as any other model. Next, we verify that most embedding models are highly compositional, while BERT shows much poorer compositionality. We verify and visualize our findings with a synthetic dataset consisting of fully transparent adjective-noun compositions. Overall, we present a thorough investigation of compositionality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。