探究大模型如何理解关系语义，发现其推理失败主因是生成顺序偏见。
Memorization, Emergence, and Explaining Reversal Failures: A Controlled Study of Relational Semantics in LLMs
- 构建基于知识图谱的合成数据，控制训练关系三元组。
- 少量逻辑监督下，浅层模型即出现关系语义的突现式学习。
- 反转任务失败主要由自回归顺序偏差导致，非语义缺失。
自回归大模型在需要通过关系词（如父亲/儿子、朋友）连接实体的关系任务中表现良好，但尚不清楚它们是否真正掌握了关系的逻辑语义（如对称性与逆向逻辑），以及反转类失败是源于关系语义缺失还是自左至右生成顺序的偏见。本文提出一种基于知识图谱的可控合成框架，生成对称/逆关系三元组文本，从零训练GPT风格自回归模型，并评估其记忆能力、逻辑推理与未见实体的上下文泛化能力。研究发现，在足够逻辑监督下，关系语义会经历明显的相变式涌现，即使在2-3层浅层模型中也能实现，且成功泛化与中间层信号的稳定性相关。此外，顺序匹配的正向/反向测试及扩散基线表明，反转失败主要由自回归顺序偏见驱动，而非逆向语义不足。
原文摘要 · Abstract (English)
Autoregressive LLMs perform well on relational tasks that require linking entities via relational words (e.g., father/son, friend), but it is unclear whether they learn the logical semantics of such relations (e.g., symmetry and inversion logic) and, if so, whether reversal-type failures arise from missing relational semantics or left-to-right order bias. We propose a controlled Knowledge Graph-based synthetic framework that generates text from symmetric/inverse triples, train GPT-style autoregressive models from scratch, and evaluate memorization, logical inference, and in-context generalization to unseen entities to address these questions. We find a sharp phase transition in which relational semantics emerge with sufficient logic-bearing supervision, even in shallow (2-3 layer) models, and that successful generalization aligns with stable intermediate-layer signals. Finally, order-matched forward/reverse tests and a diffusion baseline indicate that reversal failures are primarily driven by autoregressive order bias rather than deficient inversion semantics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。