arXiv:2605.05365cs.AIcs.CL2026-05被引 1

8B参数的推理模型,用小规模激活参数达到顶级水平。

ZAYA1-8B Technical Report

论文配图:ZAYA1-8B Technical Report
图 1 · 摘自论文原文
  • 基于MoE++架构,全程在AMD平台从零训练推理模型。
  • 数学和编码任务上媲美更大模型,AIME'25达91.9%正确率。
  • 提出递归推理方法Markovian RSA,仅传4K token仍保持高效。

我们提出ZAYA1-8B,一个以推理为核心的混合专家(MoE)模型,拥有7亿活跃参数和80亿总参数,基于Zyphra的MoE++架构构建。其预训练、中段训练及监督微调均在完整的AMD计算、网络与软件平台上完成。仅使用不到10亿活跃参数,ZAYA1-8B在多个挑战性数学与编程基准上表现媲美或超越DeepSeek-R1-0528,且在显著更大的开源推理模型中保持竞争力。该模型从零开始专为推理设计,自预训练阶段即引入推理数据,并采用答案保留截断策略。后训练采用四阶段强化学习级联:数学与谜题推理预热;400任务的RLVE-Gym课程;结合测试时计算痕迹与竞赛编程参考构建的合成代码环境进行数学与代码强化学习;以及用于对话与指令遵循的行为强化学习。我们还引入了马尔可夫式RSA,一种测试时计算方法,通过递归聚合并行推理轨迹,同时每轮仅传递有限长度的推理尾部。在TTC评估中,马尔可夫式RSA使ZAYA1-8B在AIME'25上达到91.9%,在HMMT'25上达到89.6%,仅携带4000词元尾部,缩小了与Gemini-2.5 Pro、DeepSeek-V3.2和GPT-5-High等大型模型的差距。

原文摘要 · Abstract (English)

We present ZAYA1-8B, a reasoning-focused mixture-of-experts (MoE) model with 700M active and 8B total parameters, built on Zyphra's MoE++ architecture. ZAYA1-8B's core pretraining, midtraining, and supervised fine-tuning (SFT) were performed on a full-stack AMD compute, networking, and software platform. With under 1B active parameters, ZAYA1-8B matches or exceeds DeepSeek-R1-0528 on several challenging mathematics and coding benchmarks, and remains competitive with substantially larger open-weight reasoning models. ZAYA1-8B was trained from scratch for reasoning, with reasoning data included from pretraining onward using an answer-preserving trimming scheme. Post-training uses a four-stage RL cascade: reasoning warmup on math and puzzles; a 400-task RLVE-Gym curriculum; math and code RL with test-time compute traces and synthetic code environments built from competitive-programming references; and behavioral RL for chat and instruction following. We also introduce Markovian RSA, a test-time compute method that recursively aggregates parallel reasoning traces while carrying forward only bounded-length reasoning tails between rounds. In TTC evaluation, Markovian RSA raises ZAYA1-8B to 91.9\% on AIME'25 and 89.6\% on HMMT'25 while carrying forward only a 4K-token tail, narrowing the gap to much larger reasoning models including Gemini-2.5 Pro, DeepSeek-V3.2, and GPT-5-High.

推理模型MoE架构强化学习数学推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。