Wiola是全新设计的小型语言模型架构，性能优于同类模型。
The Wiola Architecture for Efficient Small Language Models
- 从零构建五项原创组件，不沿用现有模型结构
- 1.5B参数版本在多个任务上超越LLaMA-2和Mistral
- 适合资源受限场景的高效推理，兼容HuggingFace生态
我们提出Wiola，一种完全原创的小型语言模型（SLM）架构，其结构与GPT、LLaMA、Mistral或Falcon等现有模型家族无任何继承关系。Wiola引入五项独立创新：(i) 螺旋旋转位置编码（SRPE），将令牌位置嵌入三维螺旋流形，融合绝对、相对与层级位置信号；(ii) 门控跨层注意力（GCLA），为每个解码器层提供对前两层压缩摘要的软跨注意力访问，增强层间一致性；(iii) 自适应标记合并（ATM），在中间网络层动态合并语义冗余的相邻标记，降低注意力复杂度且无信息损失；(iv) 双流前馈（DSFF），以两个并行流替换传统MLP，通过学习的逐维门控融合；(v) WiolaRMSNorm，一种引入逐维可学习偏移向量的归一化方法，防止表征崩溃。我们提供了完整的数学推导、架构图、复杂度分析及与GPT-2、LLaMA-2和Mistral的系统性对比。Wiola发布四个尺寸（120M、360M、700M、1.5B参数），完全兼容HuggingFace Transformers生态，所有22个架构单元测试均通过。
原文摘要 · Abstract (English)
We present Wiola, a fully original Small Language Model (SLM) architecture built from first principles, sharing no structural lineage with any existing model family including GPT, LLaMA, Mistral, or Falcon. Wiola introduces five independently novel components: (i) Spiral Rotary Positional Encoding (SRPE), which embeds token positions on a three-dimensional helical manifold combining absolute, relative, and hierarchical positional signals; (ii) Gated Cross-Layer Attention (GCLA), providing each decoder layer with soft cross-attention access to compressed summaries of two preceding layers for inter-layer coherence; (iii) Adaptive Token Merging (ATM), which dynamically merges se mantically redundant adjacent tokens in middle network layers to reduce attention complexity without information loss; (iv) Dual Stream Feed-Forward (DSFF), replacing the conventional MLP with two parallel streams fused by a learned per-dimension gate; and (v) WiolaRMSNorm, a modified normalisation introducing a per-dimension learned offset vector that prevents representation collapse. We provide complete mathematical derivations, architectural block diagrams, complexity analyses, and systematic comparisons against GPT-2, LLaMA-2, and Mistral. Wiola is released in four sizes (120M, 360M, 700M, and 1.5B parameters) and is fully compatible with the HuggingFace Transformers ecosystem, with all 22 architectural unit tests passing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。