大模型在重复规则任务中会突然失效，无法可靠计数。
Language models fail at extended rule following
- 用重复字符计数测试发现模型有语法敏感的计数上限
- 所有测试模型均在阈值后崩溃，且大模型也无改善
- 适合关注智能体可靠性与模型机制研究者阅读
大型语言模型虽能处理长上下文信息并回答复杂问题，但在代理类任务中还需精确保持状态并持续应用规则。我们发现该可靠性在现有模型中完全缺失。通过让126个主流模型变体对长串重复字符进行计数，发现它们均存在一个依赖模型、受语法影响的计数容量阈值，超过后即出现突然失败。这种失败不随模型规模增大、推理时间延长或使用外部工具而缓解。机制探查表明，模型仅用有限内部状态模拟计数规则，一旦耗尽即失效。这些状态也是完成更复杂任务的基础。结果表明，实现真正可靠的规则遵循需根本性新架构。
原文摘要 · Abstract (English)
Large language models are highly capable of answering difficult questions by retrieving, recombining, and attending to information in long contexts. For agentic tasks, an additional capability is required: the preservation of an exact state while repeatedly applying rules. We find that this reliability is absent across language models. To demonstrate, we query 126 leading model variants with the task of counting a long string of repeated characters, and we find they all cannot accurately count above a model-dependent, syntax-sensitive counting capacity threshold. Failures are abrupt and persist even with increasing model size, inference time computation, and external tool. Mechanistic probing indicates that models use a finite number of internal states to mimic counting as a rule and fail once these states are exhausted. Furthermore, such states are the basis for performing complex tasks beyond counting. These results indicate that fundamentally new model architectures are required for autonomous agents to achieve truly reliable rule following capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。