测试大模型能否像人一样逐步构建认知，发现它们缺乏真正的动态推理能力。
Beyond Pattern Recognition: Probing Mental Representations of LMs
- 分步给信息，观察模型如何逐步更新内部认知
- 在数学任务上，不同规模模型均无法形成有效心智表征
- 揭示大模型更多依赖模式识别而非真实推理过程
语言模型（LMs）在解决复杂推理任务方面表现出色，尤其是通过提示生成中间推理步骤时。然而，这些中间推理链究竟是动态演化的思维过程，还是仅反映大规模预训练中习得的复杂模式识别，仍是未解之谜。受人类认知启发——即随着新信息的吸收，内部模型持续更新——我们提出一种新方法来探究不同语言模型的心智表征机制。该方法通过逐步提供问题细节，使每个新信息都能基于前序内容构建并修正模型对任务的内部表征。我们在文本和图文模态下，系统比较了这种分步心智建模策略与传统全量提示方法。在MathWorld数据集上的实验表明，无论模型大小或问题复杂度如何，纯文本大模型与多模态大模型均难以建立有效的内在心智表征，这质疑了其内部认知过程的本质。
原文摘要 · Abstract (English)
Language Models (LMs) have demonstrated impressive capabilities in solving complex reasoning tasks, particularly when prompted to generate intermediate explanations. However, it remains an open question whether these intermediate reasoning traces represent a dynamic, evolving thought process or merely reflect sophisticated pattern recognition acquired during large scale pre training. Drawing inspiration from human cognition, where reasoning unfolds incrementally as new information is assimilated and internal models are continuously updated, we propose to delve deeper into the mental model of various LMs. We propose a new way to assess the mental modeling of LMs, where they are provided with problem details gradually, allowing each new piece of data to build upon and refine the model's internal representation of the task. We systematically compare this step by step mental modeling strategy with traditional full prompt methods across both text only and vision and text modalities. Experiments on the MathWorld dataset across different model sizes and problem complexities confirm that both text-based LLMs and multimodal LMs struggle to create mental representations, questioning how their internal cognitive processes work.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。