用专家混合流匹配加速语言模型生成，三步完成质量媲美传统方法。
Towards Faster Language Model Inference Using Mixture-of-Experts Flow Matching
- 将复杂潜空间流分解为局部专用向量场，提升对异质分布的建模能力。
- 新模型YAN仅需3次采样步骤，生成质量与自回归和扩散模型相当。
- 相比自回归模型快40倍，比扩散模型快1000倍，适合高效生成场景。
流匹配在保持扩散模型生成质量的同时实现显著更快的推理，是生成建模的有力范式。然而，应用于语言建模时，其在表示具有不规则几何结构（如各向异性、多模态）的复杂潜空间分布方面存在根本局限。为此，我们提出混合专家流匹配（MoE-FM）框架，通过将复杂的全局传输几何分解为局部专用向量场来捕捉潜空间中的复杂结构。基于MoE-FM，我们构建了非自回归（NAR）语言建模方法YAN，采用Transformer和Mamba架构实现。在多个下游任务中，YAN生成质量与自回归（AR）及基于扩散的NAR语言模型相当，同时仅需3次采样步骤。相比AR基线提速40倍，相比扩散语言模型最高提速1000倍，充分展现了语言建模中的巨大效率优势。
原文摘要 · Abstract (English)
Flow matching retains the generation quality of diffusion models while enabling substantially faster inference, making it a compelling paradigm for generative modeling. However, when applied to language modeling, it exhibits fundamental limitations in representing complex latent distributions with irregular geometries, such as anisotropy and multimodality. To address these challenges, we propose a mixture-of-experts flow matching (MoE-FM) framework, which captures complex global transport geometries in latent space by decomposing them into locally specialized vector fields. Building on MoE-FM, we develop a non-autoregressive (NAR) language modeling approach, named YAN, instantiated with both Transformer and Mamba architectures. Across multiple downstream tasks, YAN achieves generation quality on par with both autoregressive (AR) and diffusion-based NAR language models, while requiring as few as three sampling steps. This yields a $40\times$ speedup over AR baselines and up to a $10^3\times$ speedup over diffusion language models, demonstrating substantial efficiency advantages for language modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。