提出可完整建模长期动作序列的递归架构，解决注意力模型在长程任务中的失效问题。
Recurrence-Complete Frame-based Action Models
- 设计递归完备架构，确保能正确处理长时间依赖任务
- 训练时损失随序列长度呈幂律下降，参数量固定
- 适合需要长期规划的智能体系统，如代码生成代理
近年来，类注意力机制在大语言模型中取得巨大成功，极大释放了模型的扩展潜力。'Attention Is All You Need' 声称无需循环神经网络单元即可实现高效建模。本文对此提出质疑：现有证明表明，前向或后向计算完全并行的架构无法表示某些对长期运行智能体任务至关重要的问题类别。我们进一步推测一个临界时间 t，超过该时间后，非递归完备模型将无法正确聚合输入，这对智能体系统（如软件工程代理）有明确影响。为此，我们提出一种递归完备架构，并在基于 GitHub 的动作序列上进行训练。训练序列越长，损失遵循幂律下降，而参数量保持不变。此外，更长序列的训练始终能摊销其线性增长的耗时成本，在相同时间内获得更低的损失。
原文摘要 · Abstract (English)
In recent years, attention-like mechanisms have been used to great success in the space of large language models, unlocking scaling potential to a previously unthinkable extent. "Attention Is All You Need" famously claims RNN cells are not needed in conjunction with attention. We challenge this view. In this paper, we point to existing proofs that architectures with fully parallelizable forward or backward passes cannot represent classes of problems specifically interesting for long-running agentic tasks. We further conjecture a critical time t beyond which non-recurrence-complete models fail to aggregate inputs correctly, with concrete implications for agentic systems (e.g., software engineering agents). To address this, we introduce a recurrence-complete architecture and train it on GitHub-derived action sequences. Loss follows a power law in the trained sequence length while the parameter count remains fixed. Moreover, longer-sequence training always amortizes its linearly increasing wall-time cost, yielding lower loss as a function of wall time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。