对比三种推理方式,发现压缩循环模型受限于内存大小
Chain-of-Thought and Compressed Looped Transformers: A Memory-Budget Separation

- 用压缩循环、完整序列循环和思维链三种记忆机制比较推理能力
- 小状态循环即使运行多次也无法扩展记忆,无法解决部分复杂问题
- 适合研究推理模型内存限制的学者,尤其关注计算与记忆平衡
思维链提示和循环Transformer都在测试时增加计算量,但记忆方式不同:思维链将中间状态存储在生成的标记中并保留在上下文中,而循环Transformer通过递归隐藏激活传递状态。我们认为,这种持续可变的记忆是测试时推理的核心资源。我们比较了三种记忆模式——压缩潜在循环、完整序列状态循环和思维链临时工作区。主要结果表明,压缩循环受其递归状态大小限制。即使循环运行更长时间,也仅增加计算量,不会形成可扩展的临时工作区,因此小状态循环始终是低内存推理器,即使运行多步也无法解决经对数空间归约后为P-完全的问题,而多项式长度的思维链可以实现。该差异仅针对压缩循环;完整序列状态循环在每个输入位置保留状态,处于接近显式临时工作区的高内存模式。受控指针追踪和关联回忆遍历实验验证了这一内存预算视角,性能取决于持久状态预算是否匹配任务所需的工作内存。
原文摘要 · Abstract (English)
Chain-of-thought prompting and looped Transformers both give a fixed model more test-time computation, but they differ in what they remember. Chain-of-thought stores intermediate state in generated tokens that remain in the context, whereas a looped Transformer carries state through recurrent hidden activations. We argue that this persistent mutable memory is a central resource for test-time reasoning. We compare three memory regimes, the compressed latent loop, the full sequence-state loop, and the chain-of-thought scratchpad. Our main result shows that a compressed loop is limited by the size of its recurrent state. Running the loop longer adds computation but does not by itself create a growing scratchpad, so a loop with a small recurrent state remains a small-space reasoner even when run for many steps. Under a standard complexity assumption, such loops cannot decide problems that are P-complete under logspace reductions, whereas polynomial-length chain-of-thought can. The separation is specific to compressed loops, as full sequence-state loops carry state at every input position and live in a memory-rich regime closer to explicit scratchpads. Controlled pointer-chasing and associative-recall sweeps illustrate this memory-budget view, with performance sensitive to whether the persistent-state budget matches the task's working-memory demand.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。