一个统一框架,让视频音频生成任务自由组合,灵活处理多种输入输出。
Vorch-Omni: Multi-Task Orchestration of Sight and Sound

- 用条件掩码和任务标识区分输入输出,支持10种以上多模态生成任务。
- 通过视觉语言模型与视频自编码器双路径,精准捕捉视觉语义信息。
- 适合需要跨模态生成、编辑或扩展的创作者与研究者使用。
近年来生成视频建模的进步实现了多样化的生成、参考式合成、内容扩展与编辑,但现有方法多依赖碎片化的任务专用模型。通用模型需区分异构的目标、源信号与参考信号,以决定生成、保留或引导的内容,同时降低任务间的干扰。跨模态音视频联合生成进一步加剧挑战,因模态间存在多样的条件与输出配置。我们提出 Vorch-Omni,一种基于任意条件到任意输出形式的统一多任务音视频合成框架。该框架灵活地将视频与音频信号视为条件输入或生成目标。通过词元级条件掩码与任务标识符,区分目标、源内容与参考信息;位置类型则分离时间上下文与独立条件。为捕捉语义与结构信息,Vorch-Omni采用互补的视觉条件路径:视觉语言模型解析采样帧并结合文本指令,视频变分自编码器(video VAE)将条件编码为潜在词元以直接引导生成。我们还构建了分布式数据流水线,用于整理多样化的时序对齐音视频片段,生成结构化描述与元数据,并平衡异构任务分布。基于单一流匹配扩散变换器,无需任务特异性架构调整,支持超过10种任务,包括文本到视频、文本到音视频、图像与参考条件生成、时间扩展、音频驱动生成、视频转换及音视频编辑。该统一框架为通用音视频生成与操作提供了可扩展的基础。
原文摘要 · Abstract (English)
Recent advances in generative video modeling have enabled diverse generation, reference-based synthesis, extension, and editing, but existing approaches often rely on fragmented task-specific models. A general model must distinguish heterogeneous target, source, and reference signals to determine what to generate, preserve, or use as guidance, while reducing interference among tasks. Joint audio-visual generation further increases this challenge by introducing diverse conditioning and output configurations across modalities. We present Vorch-Omni, a unified multi-task framework for audio-visual synthesis based on an arbitrary-condition-to-arbitrary-output formulation. It flexibly treats video and audio signals as either conditioning inputs or generation targets. Token-level conditioning masks and task identifiers distinguish targets, source content, and references, while position types separate temporal context from independent conditions. To capture semantic and structural information, Vorch-Omni employs complementary visual conditioning pathways: a vision-language model interprets sampled frames with text instructions, and a video VAE encodes conditions into latent tokens for direct guidance. We further build a distributed data pipeline to curate diverse temporally aligned audio-visual clips, generate structured captions and metadata, and balance heterogeneous task distributions. Built on a single flow-matching diffusion transformer without task-specific architectural changes, Vorch-Omni supports over 10 tasks, including text-to-video, text-to-audio-video, image- and reference-conditioned generation, temporal extension, audio-driven generation, video transformation, and audio-visual editing. This unified framework provides a scalable foundation for general-purpose audio-visual generation and manipulation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。