arXiv:2509.17765cs.CLcs.AI2025-09被引 450

通义千问3-Omni首次实现多模态全场景性能领先,语音表现尤为突出。

Qwen3-Omni Technical Report

  • 采用思维者-说话者MoE架构,统一处理文本、图像、音频和视频输入
  • 在36个音视频基准上32项开源SOTA,22项整体SOTA,优于GPT-4o等闭源模型
  • 支持119种语言文本交互,首包延迟低至234毫秒,适合实时语音应用

我们提出Qwen3-Omni,首个在文本、图像、音频和视频四大模态上均保持顶尖性能的单一多模态模型,且不逊于同规模单模态模型。Qwen3-Omni在36个音视频基准中达成32项开源SOTA、22项整体SOTA,超越Gemini-2.5-Pro、Seed-ASR及GPT-4o-Transcribe等闭源模型。其采用Thinker-Talker MoE架构,统一感知与生成,实现流畅文本与自然实时语音。支持119种语言文本交互、19种语言语音理解、10种语言语音生成。通过多码本方案与轻量因果卷积网络,将块级扩散替换为流式合成,冷启动下理论端到端首包延迟仅234毫秒。引入显式思维模块增强跨模态推理能力。针对缺乏通用音频字幕模型的问题,微调得到Qwen3-Omni-30B-A3B-Captioner,可生成高保真低幻觉音频描述。相关模型已开源,许可协议为Apache 2.0。

原文摘要 · Abstract (English)

We present Qwen3-Omni, a single multimodal model that, for the first time, maintains state-of-the-art performance across text, image, audio, and video without any degradation relative to single-modal counterparts. Qwen3-Omni matches the performance of same-sized single-modal models within the Qwen series and excels particularly on audio tasks. Across 36 audio and audio-visual benchmarks, Qwen3-Omni achieves open-source SOTA on 32 benchmarks and overall SOTA on 22, outperforming strong closed-source models such as Gemini-2.5-Pro, Seed-ASR, and GPT-4o-Transcribe. Qwen3-Omni adopts a Thinker-Talker MoE architecture that unifies perception and generation across text, images, audio, and video, yielding fluent text and natural real-time speech. It supports text interaction in 119 languages, speech understanding in 19 languages, and speech generation in 10 languages. To reduce first-packet latency in streaming synthesis, Talker autoregressively predicts discrete speech codecs using a multi-codebook scheme. Leveraging the representational capacity of these codebooks, we replace computationally intensive block-wise diffusion with a lightweight causal ConvNet, enabling streaming from the first codec frame. In cold-start settings, Qwen3-Omni achieves a theoretical end-to-end first-packet latency of 234 ms. To further strengthen multimodal reasoning, we introduce a Thinking model that explicitly reasons over inputs from any modality. Since the research community currently lacks a general-purpose audio captioning model, we fine-tuned Qwen3-Omni-30B-A3B to obtain Qwen3-Omni-30B-A3B-Captioner, which produces detailed, low-hallucination captions for arbitrary audio inputs. Qwen3-Omni-30B-A3B, Qwen3-Omni-30B-A3B-Thinking, and Qwen3-Omni-30B-A3B-Captioner are publicly released under the Apache 2.0 license.

多模态语音生成MoE架构实时推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。