统一生成任意模态的自回归模型，支持文本、图像、语音无缝输出。
AR-Omni: A Unified Autoregressive Model for Any-to-Any Generation
- 用单一Transformer解码器实现跨模态生成，无需额外专家模块。
- 语音生成实时因子达0.88，图像保真度提升，三模态质量均强。
- 适合需要多模态统一生成的系统开发者与研究者。
现实世界的感知与交互本质上是多模态的，包含语言、视觉和语音，这推动了支持多模态输入与输出的“全模态”多模态大模型（Omni MLLMs）的发展。尽管一系列全模态模型已出现，但多数系统仍依赖额外专家组件实现多模态生成，限制了统一训练与推理的简洁性。自回归（AR）建模凭借单一词元流、统一的下一个词元预测目标和单一解码器，在文本领域展现出优雅与可扩展性。受此启发，我们提出AR-Omni：一种基于自回归范式的统一任意到任意生成模型，无需任何专家解码器。该模型支持自回归文本生成、图像生成及流式语音生成，全部基于单一Transformer解码器。我们进一步解决统一自回归建模中的三个实际问题：通过任务感知损失重加权缓解模态不平衡；引入轻量级逐标记感知对齐损失提升图像保真度；采用有限状态解码机制平衡稳定性与创造性。实验表明，AR-Omni在三模态上均达到优异质量，且保持实时性能，语音生成实时因子为0.88。
原文摘要 · Abstract (English)
Real-world perception and interaction are inherently multimodal, encompassing not only language but also vision and speech, which motivates the development of "Omni" MLLMs that support both multimodal inputs and multimodal outputs. While a sequence of omni MLLMs has emerged, most existing systems still rely on additional expert components to achieve multimodal generation, limiting the simplicity of unified training and inference. Autoregressive (AR) modeling, with a single token stream, a single next-token objective, and a single decoder, is an elegant and scalable foundation in the text domain. Motivated by this, we present AR-Omni, a unified any-to-any model in the autoregressive paradigm without any expert decoders. AR-Omni supports autoregressive text and image generation, as well as streaming speech generation, all under a single Transformer decoder. We further address three practical issues in unified AR modeling: modality imbalance via task-aware loss reweighting, visual fidelity via a lightweight token-level perceptual alignment loss for image tokens, and stability-creativity trade-offs via a finite-state decoding mechanism. Empirically, AR-Omni achieves strong quality across three modalities while remaining real-time, achieving a 0.88 real-time factor for speech generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。