通过细胞自动机测试大模型推理深度极限，发现递归与记忆可提升但仍有瓶颈。
Beyond Memorization: Extending Reasoning Depth with Recurrence, Memory and Test-Time Compute Scaling
- 用一维细胞自动机隔离记忆干扰，聚焦多步推理能力
- 模型在超过3步推理时准确率骤降，表明深度扩展困难
- 递归、记忆和测试时计算能延展推理，但效果受限于架构
推理是大语言模型的核心能力，但其多步推理的学习与执行机制仍不明确。本文在可控的一维细胞自动机（1dCA）框架下研究该问题，通过使用分离的训练与测试规则排除记忆干扰。给定一段短状态序列，模型需推断隐藏局部规则并链式预测多个未来步骤。评估显示，大多数大模型难以可靠解决自然语言代理任务。实验发现，从零训练的神经网络虽可学习规则推断并达到高单步预测准确率，但随着中间推理步骤增加，性能急剧下降。增加模型深度至关重要，而通过递归、记忆或测试时计算扩展有效深度虽有改善，但仍存在上限。代码已开源：https://github.com/RodkinIvan/associative-recurrent-memory-transformer/tree/ACT
原文摘要 · Abstract (English)
Reasoning is a core capability of large language models, yet how multi-step reasoning is learned and executed remains unclear. We study this question in a controlled cellular-automata (1dCA) framework that excludes memorisation by using disjoint training and test rules. Given a short state sequence, the model is required to infer the hidden local rule and then chain it to predict multiple future steps. Our evaluation shows that LLMs largely fail to reliably solve a natural-language proxy of the proposed task. We find that most neural architectures trained from scratch can learn rule inference and achieve high next-step accuracy, but performance drops sharply as the required number of intermediate reasoning steps increases. Experiments show that increasing model depth is crucial, and extending effective depth via recurrence, memory, or test-time compute improves results but remains bounded. The code is available on github: https://github.com/RodkinIvan/associative-recurrent-memory-transformer/tree/ACT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。