发现大模型词嵌入具有共享的全局与局部几何结构，可用于跨模型向量转换。
Shared Global and Local Geometry of Language Model Embeddings
- 分析词嵌入的相对方向与局部线性结构，揭示跨模型几何相似性。
- 嵌入位于低维流形上，低内在维度的词聚类语义更清晰。
- 提出EMB2EMB方法，实现不同模型间转向向量的线性转换。
近期研究指出模型可能共享共同表示。本文发现大型语言模型的词嵌入存在大量几何相似性。首先，我们观察到‘全局’相似性：词嵌入常具有相似的相对方向。其次，通过两种方式刻画局部几何：(1) 使用局部线性嵌入（Locally Linear Embeddings），(2) 定义每项嵌入的内在维度简单度量。这两种方法均能发现词嵌入间的局部相似性。此外，内在维度表明嵌入位于低维流形上，内在维度较低的词通常形成语义连贯的簇，而较高者则不然。基于此发现，我们提出EMB2EMB，一种简单的线性变换方法，可将一个语言模型的转向向量映射至另一维度不同的模型。
原文摘要 · Abstract (English)
Researchers have recently suggested that models share common representations. In our work, we find numerous geometric similarities across the token embeddings of large language models. First, we find ``global'' similarities: token embeddings often share similar relative orientations. Next, we characterize local geometry in two ways: (1) by using Locally Linear Embeddings, and (2) by defining a simple measure for the intrinsic dimension of each embedding. Both characterizations allow us to find local similarities across token embeddings. Additionally, our intrinsic dimension demonstrates that embeddings lie on a lower dimensional manifold, and that tokens with lower intrinsic dimensions often have semantically coherent clusters, while those with higher intrinsic dimensions do not. Based on our findings, we introduce EMB2EMB, a simple application to linearly transform steering vectors from one language model to another, despite the two models having different dimensions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。