小模型新架构,三组件提升效率与长程建模能力
Wiola 13M, a Gated Spiral Attention Architecture for Parameter Efficient Small Language Models

- 螺旋位置编码+门控注意力+蝴蝶前馈,每层仅增微小开销
- 在1300万参数下实现接近大模型的长距离依赖捕捉能力
- 适合移动端部署、快速实验和可复现的小模型研究
十到一亿参数范围内的小型语言模型适用于设备端推理、快速实验和受控科学探究,但多数仍沿用标准Transformer结构,未针对小规模场景优化。本文提出Wiola,一种仅含解码器的语言模型,其创新集中于每层的三个即插即用组件:第一,螺旋旋转位置编码通过逐维缓慢增长的频率扰动,使相位轨迹向外发散,增强长程区分能力且不增加参数;第二,门控螺旋注意力引入基于查询流因果累积统计的逐头内容自适应标量门,实现低开销的软头选择;第三,蝴蝶前馈模块以乘法交互和块内旁路路径替代传统扩展层,在匹配四倍门控线性单元参数量的同时改善浅层堆叠的梯度流动。我们形式化每个组件,推导精确的参数与计算预算,并证明门控注意力在全序列训练与缓存自回归解码间存在精确等价性,推理时无近似误差。此外,我们在标准小故事语料上描述了完全可复现的训练与评估协议,参考实现已开源并支持权重发布。
原文摘要 · Abstract (English)
Small language models in the ten to one hundred million parameter range are attractive for on device inference, rapid experimentation, and controlled scientific study, yet most of them reuse the standard transformer block without adaptation to the small scale regime. We present Wiola, a decoder only language model whose novelty is concentrated in three drop in components of every layer. First, Spiral Rotary Positional Encoding perturbs the standard rotary frequencies by a slowly growing per dimension factor so that phase trajectories fan outward, improving long range discrimination while adding no parameters. Second, Gated Spiral Attention introduces a per head, content adaptive scalar gate derived from a causal cumulative statistic of the query stream, providing an implicit and differentiable form of soft head selection at negligible cost. Third, the Butterfly feed forward block replaces the conventional expansion layer with a multiplicative interaction and an intra block bypass path, matching the parameter count of a four times gated linear unit block while improving gradient flow in shallow stacks. We formalize each component, derive exact parameter and computation budgets, and prove that the gated attention admits an exact and numerically verified equivalence between full sequence training and cached autoregressive decoding, so that no approximation is introduced at inference time. We also describe a fully reproducible training and evaluation protocol on a standard tiny story corpus. The reference implementation is released as an open source package with weights ready publishing support.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。