研究大模型如何学习并运用结构化知识，发现其推理能力与结构学习相关但生成仍受限。
On the Emergence and Test-Time Use of Structural Information in Large Language Models
- 基于语言结构变换构建可控数据集，验证模型对抽象结构的学习能力。
- 结构学习能力随复杂推理任务出现，但在测试时组合生成仍不理想。
- 适合关注大模型认知机制与生成局限的研究者阅读。
从观测数据中学习结构化信息是生成训练语料外新知识的核心。这在科学发现的机理理解以及测试时灵活组合生成中尤为重要。因此，我们研究语言模型如何学习抽象结构，并在测试时利用所学结构信息。为确保实验控制，我们设计了一个基于语言结构变换的自然语言数据集。实证表明，结构信息学习的出现与复杂推理任务相关，但模型在测试时进行组合生成的能力依然有限。
原文摘要 · Abstract (English)
Learning structural information from observational data is central to producing new knowledge outside the training corpus. This holds for mechanistic understanding in scientific discovery as well as flexible test-time compositional generation. We thus study how language models learn abstract structures and utilize the learnt structural information at test-time. To ensure a controlled setup, we design a natural language dataset based on linguistic structural transformations. We empirically show that the emergence of learning structural information correlates with complex reasoning tasks, and that the ability to perform test-time compositional generation remains limited.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。