探究大模型能否像人一样进行类比推理,发现其有潜力但仍有局限。
The Curious Case of Analogies: Investigating Analogical Reasoning in Large Language Models
- 通过比例和故事类比测试模型关系编码能力
- 正确推理时关系信息在深层传播,失败则缺失关键关系
- 补全特定位置隐藏表示可部分提升新情境应用能力
类比推理是人类认知的核心,支撑多种智力活动。尽管先前研究显示大语言模型能表征任务模式和表层概念,但它们是否能编码高层级关系并应用于新情境仍不明确。本文通过比例类比和故事类比,揭示三个关键发现:第一,模型能有效编码类比实体间的潜在关系;正确案例中属性与关系信息在中间至高层传播,而推理失败反映这些层中关系信息缺失。第二,与人类不同,模型不仅在关系缺失时出错,尝试将关系迁移到新实体时也常失败;此时在关键标记位置策略性修补隐藏表示,可在一定程度上促进信息传递。第三,模型成功类比表现为相似情境间强结构对齐,失败则体现对齐退化或错位。总体而言,模型展现出新兴但有限的高阶关系编码与应用能力,揭示了与人类认知的异同。
原文摘要 · Abstract (English)
Analogical reasoning is at the core of human cognition, serving as an important foundation for a variety of intellectual activities. While prior work has shown that LLMs can represent task patterns and surface-level concepts, it remains unclear whether these models can encode high-level relational concepts and apply them to novel situations through structured comparisons. In this work, we explore this fundamental aspect using proportional and story analogies, and identify three key findings. First, LLMs effectively encode the underlying relationships between analogous entities; both attributive and relational information propagate through mid-upper layers in correct cases, whereas reasoning failures reflect missing relational information within these layers. Second, unlike humans, LLMs often struggle not only when relational information is missing, but also when attempting to apply it to new entities. In such cases, strategically patching hidden representations at critical token positions can facilitate information transfer to a certain extent. Lastly, successful analogical reasoning in LLMs is marked by strong structural alignment between analogous situations, whereas failures often reflect degraded or misplaced alignment. Overall, our findings reveal that LLMs exhibit emerging but limited capabilities in encoding and applying high-level relational concepts, highlighting both parallels and gaps with human cognition.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。