大模型需提示才能计数,自发计数能力仍不足。
Sequential Enumeration in Large Language Models
- 通过提示引导,部分大模型可执行系统性计数
- 无提示时所有模型均无法自发计数
- 揭示神经与符号系统在组合泛化上的根本差异
可靠计数和生成序列仍是神经网络(包括大语言模型)的重大挑战。尽管规则系统可通过串行计算轻松处理,但神经模型需通过学习获得此类能力。先前研究显示循环结构仅能近似追踪序列,现代深度学习系统能否对离散符号序列部署系统性计数仍不明确。本文探究五种先进大模型(含专有、开源及推理模型)的序列枚举能力,采用多种提示策略考察思维链对计数策略自发出现的作用。同时评估同架构不同规模的开源模型,检验计数能力是否遵循缩放规律,并分析计数过程中的嵌入动态以研究数量编码的涌现。结果表明:仅在明确提示下,部分模型可部署计数程序;但若仅要求列举序列中项目数,所有模型均不会自发计数。这说明尽管大模型具备惊人涌现能力,仍无法稳健系统地运用计数程序,凸显神经与符号方法在组合泛化上的持续差距。
原文摘要 · Abstract (English)
Reliably counting and generating sequences of items remain a significant challenge for neural networks, including Large Language Models (LLMs). Indeed, although this capability is readily handled by rule-based symbolic systems based on serial computation, learning to systematically deploy counting procedures is difficult for neural models, which should acquire these skills through learning. Previous research has demonstrated that recurrent architectures can only approximately track and enumerate sequences of events, and it remains unclear whether modern deep learning systems, including LLMs, can deploy systematic counting procedures over sequences of discrete symbols. This paper aims to fill this gap by investigating the sequential enumeration abilities of five state-of-the-art LLMs, including proprietary, open-source, and reasoning models. We probe LLMs in sequential naming and production tasks involving lists of letters and words, adopting a variety of prompting instructions to explore the role of chain-of-thought in the spontaneous emerging of counting strategies. We also evaluate open-source models with the same architecture but increasing size to see whether the mastering of counting principles follows scaling laws, and we analyze the embedding dynamics during sequential enumeration to investigate the emergent encoding of numerosity. We find that some LLMs are indeed capable of deploying counting procedures when explicitly prompted to do so, but none of them spontaneously engage in counting when simply asked to enumerate the number of items in a sequence. Our results suggest that, despite their impressive emergent abilities, LLMs cannot yet robustly and systematically deploy counting procedures, highlighting a persistent gap between neural and symbolic approaches to compositional generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。