让状态空间模型通过调用外部工具实现任意长序列生成。
To Infinity and Beyond: Tool-Use Unlocks Length Generalization in State Space Models
- 引入外部工具增强状态空间模型，突破其长度建模限制。
- 在算术、推理和编码任务上实现无限长度泛化能力。
- 适合需要长上下文交互的智能体系统研究者参考。
状态空间模型（SSMs）已成为序列建模中替代Transformer的主流方案，其核心优势在于长上下文和长文本生成时的高效性，得益于固定大小的内存和计算复杂度的线性增长。本文首先证明了一个简单但关键的理论结果：在严格定义下，SSMs无法准确解决任何“真正长格式”的生成问题，这削弱了其主要竞争优势。然而，我们进一步表明，通过赋予SSMs与外部工具的交互能力，可有效缓解此限制。事实上，在合适的工具访问机制和依赖任务的训练数据下，SSMs能够学习求解任意可处理的问题，并实现对任意长度/复杂度的泛化（即长度泛化）。基于该理论发现，我们实证展示了工具增强型SSMs在多种算术、推理和编码任务中表现出卓越的长度泛化性能。这些结果表明，SSMs在交互式工具驱动和智能体场景中，有望成为Transformer的高效替代方案。
原文摘要 · Abstract (English)
State Space Models (SSMs) have become the leading alternative to Transformers for sequence modeling. Their primary advantage is efficiency in long-context and long-form generation, enabled by fixed-size memory and linear scaling of computational complexity. We begin this work by showing a simple theoretical result stating that SSMs cannot accurately solve any ``truly long-form'' generation problem (in a sense we formally define), undermining their main competitive advantage. However, we show that this limitation can be mitigated by allowing SSMs interactive access to external tools. In fact, we show that given the right choice of tool access and problem-dependent training data, SSMs can learn to solve any tractable problem and generalize to arbitrary problem length/complexity (i.e., achieve length generalization). Following our theoretical finding, we demonstrate that tool-augmented SSMs achieve remarkable length generalization on a variety of arithmetic, reasoning, and coding tasks. These findings highlight SSMs as a potential efficient alternative to Transformers in interactive tool-based and agentic settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。