发现大模型关系解码器本质是共性语义属性的提取，可高效压缩。
The Structure of Relation Decoding Linear Operators in Large Language Models
- 用张量网络压缩多个关系解码器，精度损失小。
- 解码器实际提取的是'国家属地'等共性属性，而非具体关系。
- 解释了为何模型只能泛化到语义相近的新关系。
本文研究了Hernandez等人[2023]提出的在Transformer语言模型中解码特定关系事实的线性算子结构。将单关系发现扩展至多关系集合，系统描绘其组织方式。结果表明，这些关系解码器集合可通过简单的三阶张量网络大幅压缩，且解码精度基本不变。为解释这一意外冗余，我们设计交叉评估协议：将每个线性解码算子应用于其他所有关系的主体。结果显示，这些线性映射并未编码独立关系，而是提取重复出现的粗粒度语义属性（如'首都所属国家'与'食物所属国家'均属于'国家-某物'属性）。这种以属性为中心的结构既解释了算子的可压缩性，也说明其泛化能力仅限于语义相近的新关系。因此，本研究将变压器语言模型中的线性关系解码解释为以属性为基础，而非针对特定关系。
原文摘要 · Abstract (English)
This paper investigates the structure of linear operators introduced in Hernandez et al. [2023] that decode specific relational facts in transformer language models. We extend their single-relation findings to a collection of relations and systematically chart their organization. We show that such collections of relation decoders can be highly compressed by simple order-3 tensor networks without significant loss in decoding accuracy. To explain this surprising redundancy, we develop a cross-evaluation protocol, in which we apply each linear decoder operator to the subjects of every other relation. Our results reveal that these linear maps do not encode distinct relations, but extract recurring, coarse-grained semantic properties (e.g., country of capital city and country of food are both in the country-of-X property). This property-centric structure clarifies both the operators' compressibility and highlights why they generalize only to new relations that are semantically close. Our findings thus interpret linear relational decoding in transformer language models as primarily property-based, rather than relation-specific.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。