发现语言模型中关系的线性特性随层和表达方式变化
Relational Linear Properties in Language Models: An Empirical Investigation
- 用KL散度探测关系的线性映射，避免粗略近似
- 不同模型、层间线性程度不一，与语言信息分布一致
- 同一关系换说法，线性特性会显著改变
线性性质在语言模型表征中普遍存在，但实验验证仍具挑战。本文聚焦关系线性：固定关系（如“演奏”）下，对象（如“小号”）的解码向量可由其主语（如“迈尔斯·戴维斯”）的编码向量通过线性映射预测。我们提出一种基于KL散度的探测方法，验证Marconato等（2025）提出的假设，并考察其在多层及改写后的关系查询中的表现。该方法比Hernandez等（2024）的线性关系嵌入更高效，避免了粗糙的雅可比近似。在四个数据集上的结果表明，关系线性在不同模型中表现各异，呈现与先前观察一致的层间模式，且受关系表述方式影响显著。
原文摘要 · Abstract (English)
Linear properties are ubiquitous in the representations of language models; however, testing them experimentally remains a challenging task. This work focuses on relational linearity: the hypothesis that, for a fixed relation (e.g., "plays"), the unembedding of an object (e.g., "trumpet") can be predicted from the embedding of its subject (e.g.,"Miles Davis") by a linear map. We present an experimental method to test the formulation of relational linearity by Marconato et al. (2025). Specifically, we introduce a probing method, based on Kullback-Leibler divergence, to evaluate this property and examine its variation across layers and paraphrased relational queries. It is also more efficient than previous work; for example, it avoids the crude Jacobian approximations used in Linear Relational Embeddings by Hernandez et al. (2024). Our findings across four datasets show that relational linearity varies across models, exhibits layer-wise patterns consistent with prior observations about linguistic information in model representations, and is differently affected by changes in how the relation is phrased.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。