大模型能凭记忆推理简单逻辑关系，但依赖训练时的隐含学习。
Out-of-Context Reasoning in Large Language Models
- 仅靠训练中接触过的逻辑规则，不给提示也能推理
- 在传递性测试中表现显著优于随机猜测
- 推理能力主要在训练阶段形成，而非推理时产生
我们研究大语言模型（LLMs）如何通过等式、不等式和包含等简单二元关系来推理记忆知识。与上下文推理不同，这些公理（如 a < b, b < c）仅在训练中出现，不在任务提示中提供（例如评估 a < c）。任务需要一步或多步推理，且涉及多源数据聚合，表现出随任务复杂度提升性能下降的趋势。我们提出一种轻量级技术——跨上下文表征学习，仅对新符号嵌入进行训练并在未见任务上评估。在自反性、对称性和传递性测试中，模型表现显著优于随机水平，可通过多种表述变体提取正确答案，但在单个查询上仍无法保持一致推理。分析表明，学习到的嵌入具有结构化组织，暗示真正的关系理解。令人惊讶的是，核心推理过程发生在训练阶段，而非推理阶段。
原文摘要 · Abstract (English)
We study how large language models (LLMs) reason about memorized knowledge through simple binary relations such as equality ($=$), inequality ($<$), and inclusion ($\subset$). Unlike in-context reasoning, the axioms (e.g., $a < b, b < c$) are only seen during training and not provided in the task prompt (e.g., evaluating $a < c$). The tasks require one or more reasoning steps, and data aggregation from one or more sources, showing performance change with task complexity. We introduce a lightweight technique, out-of-context representation learning, which trains only new token embeddings on axioms and evaluates them on unseen tasks. Across reflexivity, symmetry, and transitivity tests, LLMs mostly perform statistically significant better than chance, making the correct answer extractable when testing multiple phrasing variations, but still fall short of consistent reasoning on every single query. Analysis shows that the learned embeddings are organized in structured ways, suggesting real relational understanding. Surprisingly, it also indicates that the core reasoning happens during the training, not inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。