小模型高效推理新架构，结合谱记忆与注意力机制
Nautile-370M: Spectral Memory Meets Attention in a Small Reasoning Model
- 采用SCA与Transformer交替的混合结构，兼顾长序列效率与表达能力
- 37100万参数模型在单个TPU上训练，支持连续极限下的精确记忆读取
- 适合资源受限场景下的复杂推理任务，尤其关注响应质量优化
我们提出Nautile-370M，一个3.71亿参数的小型语言模型，旨在严格限制参数量和推理预算下实现高效推理。该模型采用混合骨干结构：两个受SeqCondenser启发的线性时间谱序列算子SCA层，与一个Transformer层交替排列。此设计旨在保留结构化序列模型的长上下文效率与状态追踪优势，同时保持注意力机制的逐标记路由表达力。模型在单个Cloud TPU v4-64 Pod切片（通过Google TPU研究云计划获得）上训练，后续强化学习阶段在单个NVIDIA DGX Spark上完成。我们证明，SCA读出机制可精确恢复前缀摘要中的任意单个标记，并可将softmax注意力输出作为特例重现，表明在连续极限下SCA至少与全自注意力一样具有表达力。我们还描述了训练数据流水线，并概述了专用于推理、验证与响应质量优化的强化学习阶段。
原文摘要 · Abstract (English)
We present Nautile-370M, a 371-million-parameter small language model designed for efficient reasoning under strict parameter and inference budgets. Nautile-370M uses a hybrid backbone in which two SeqCond Attention (SCA) layers, a linear-time spectral sequence operator inspired by SeqCondenser, alternate with one transformer layer. This design aims to retain the long-context efficiency and state-tracking benefits of structured sequential models while preserving the expressive token-to-token routing of attention. The model was trained on a single Cloud TPU v4-64 pod slice provided through the Google TPU Research Cloud (TRC) program; the subsequent reinforcement learning stage was carried out on a single NVIDIA DGX Spark. We prove that the SCA readout mechanism can exactly retrieve any individual token from the prefix summary and can reproduce any output of softmax attention as a special case, establishing that SCA is at least as expressive as full self-attention in the continuous limit. We also describe the training data pipeline and outline a reinforcement learning stage specialized for reasoning, verification, and response quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。