arXiv:2604.19052cs.CL2026-04ACL被引 2

发现大模型用细胞网格编码语篇关系,可解释其推理机制。

Cell-Based Representation of Relational Binding in Language Models

论文配图:Cell-Based Representation of Relational Binding in Language Models
图 1 · 摘自论文原文
  • 用低维线性子空间建模实体-关系绑定,每个单元对应一对索引。
  • 在多领域模型中,关系索引可线性解码,且空间呈网格结构。
  • 该表示支持跨上下文迁移,是模型进行关系推理的关键机制。

理解语篇需追踪实体及其间的关系。尽管大语言模型在关系推理上表现优异,但其如何绑定实体、关系与属性仍不明确。本文研究语篇级关系绑定,发现大模型通过一种基于细胞的绑定表示(CBR)实现:在低维线性子空间中,每个“细胞”对应一个实体-关系索引对,推理时从对应细胞提取绑定属性。通过标注实体与关系索引的受控多句数据,我们利用偏最小二乘回归从属性词激活中解码这些索引,识别出CBR子空间。在多个领域和两类模型中,索引可线性解码,且投影空间呈现网格状几何结构。进一步发现,上下文特异的CBR表示在激活空间中由平移向量关联,支持跨上下文迁移。激活修补实验表明,操控此子空间会系统性改变关系预测,扰动它会破坏性能,为模型依赖CBR进行关系绑定提供了因果证据。

原文摘要 · Abstract (English)

Understanding a discourse requires tracking entities and the relations that hold between them. While Large Language Models (LLMs) perform well on relational reasoning, the mechanism by which they bind entities, relations, and attributes remains unclear. We study discourse-level relational binding and show that LLMs encode it via a Cell-based Binding Representation (CBR): a low-dimensional linear subspace in which each ``cell'' corresponds to an entity--relation index pair, and bound attributes are retrieved from the corresponding cell during inference. Using controlled multi-sentence data annotated with entity and relation indices, we identify the CBR subspace by decoding these indices from attribute-token activations with Partial Least Squares regression. Across domains and two model families, the indices are linearly decodable and form a grid-like geometry in the projected space. We further find that context-specific CBR representations are related by translation vectors in activation space, enabling cross-context transfer. Finally, activation patching shows that manipulating this subspace systematically changes relational predictions and that perturbing it disrupts performance, providing causal evidence that LLMs rely on CBR for relational binding.

语言模型关系推理表征分析可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。