Evo统一了自回归与扩散模型,让语言生成像进化一样渐进优化。
Evo: Autoregressive-Diffusion Large Language Models with Evolving Balance
- 将文本生成看作潜在向量随时间演化的连续过程,动态平衡自回归与扩散策略。
- 8B模型在15项任务上达到顶尖或接近顶尖水平,推理速度不降反而更快。
- 适合追求高质量生成与高效推理的开发者,尤其看重逻辑推理能力者。
我们提出Evo,一种双模态潜在轨迹模型,将自回归(AR)与基于扩散的语言生成统一于连续演化生成框架中。不同于将二者视为独立范式,Evo将文本生成重构为潜在流:每个词元关联一个向量嵌入,其随进展变量 $t_i \in [0, 1]$ 演化,反映语义成熟度。低 $t_i$ 对应自信的AR式精炼,高 $t_i$ 则激发扩散式规划,使模型可根据不确定性自适应平衡。理论上,我们证明两者均是共享概率流的离散化形式,并从统一变分ELBO推导出Evo的训练目标。模型以时序条件Transformer实现,由共享向量场驱动,端到端训练以联合推断潜在码及其进展时间。解码时,Evo执行高效、语义感知的精炼,生成高质量输出且不牺牲速度。实证表明,Evo 8B在15个多样化基准测试中表现卓越,涵盖推理(GSM8K、ARC-C)、代码生成(HumanEval、MBPP)与通用语言理解,同时保持快速推理。结果表明,Evo为大模型设计带来新范式,兼具优异生成质量、鲁棒符号推理与解码效率。
原文摘要 · Abstract (English)
We introduce \textbf{Evo}, a duality latent trajectory model that bridges autoregressive (AR) and diffusion-based language generation within a continuous evolutionary generative framework. Rather than treating AR decoding and diffusion generation as separate paradigms, Evo reconceptualizes text generation as a latent flow: each token is associated with a vector-valued embedding that evolves over a progression variable $t_i \in [0, 1]$, indicating its semantic maturity. Low $t_i$ values correspond to confident AR-like refinement, while high values invoke diffusion-style planning, allowing the model to adaptively balance AR and diffusion based on uncertainty. Theoretically, we show that both AR and diffusion models emerge as discretizations of a shared probability flow, and we derive Evo's training objective from a unified variational ELBO. The model is implemented as a time-conditioned Transformer governed by a shared vector field, trained end-to-end to jointly infer latent codes and their progression times. During decoding, Evo performs efficient, semantics-aware refinement, achieving high-quality outputs without sacrificing speed. Empirically, Evo 8B achieves state-of-the-art or highly competitive results on 15 diverse benchmarks, including reasoning (GSM8K, ARC-C), code generation (HumanEval, MBPP), and general language understanding, while maintaining fast inference speed. Our results demonstrate that Evo delivers a new paradigm for LLM design with strong generation quality, robust symbolic reasoning, and decoding efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。