arXiv:2607.25948cs.CVcs.AI2026-07中稿 · ICML

用单一模型实现任意模态间的双向生成,无需特定头或损失函数。

MODUS: Decoder-Only Any-to-Any Modeling of Diverse Modalities

论文配图:MODUS: Decoder-Only Any-to-Any Modeling of Diverse Modalities
图 1 · 摘自论文原文
  • 采用解码器仅架构,对所有模态一视同仁,输入输出自由切换。
  • 在多个基准上表现接近专用模型,支持链式生成与跨模态自验证。
  • 适合需要灵活多模态生成的科研与应用,开源可复现。

任何模态到任何模态的模型可在单一网络中从任意组合的其他模态预测目标模态,该范式已用于多模态视觉与视觉-语言模型,并在生态学、天文学等科学领域逐渐普及。现有方法通常使用编码器-解码器或扩散架构从零训练,限制了性能,且无法利用强大的预训练解码器模型作为先验。本文研究解码器仅架构下的任意模态建模,使所有模态对称处理,支持任意模态作为输入或输出,无需模态特定头、损失或任务流水线。因每种模态既是输入也是输出,所提出的模型 Modus 可支持链式生成或通过生成另一模态评分自身输出等应用。Modus 展现出优异的开箱即用性能,在多个基准上与专用及多任务基线相当,仅用单一模型完成多项任务。所有材料已在 https://modus-multimodal.epfl.ch/ 开源。

原文摘要 · Abstract (English)

Any-to-any models predict any modality from any combination of others within a single network, a formulation used in multimodal vision and vision-language models, and increasingly in scientific domains such as ecology and astronomy. Existing any-to-any models are typically trained from scratch using encoder-decoder or diffusion architectures, impacting their performance and preventing them from using strong pre-trained decoder-only models as a prior. In this work, we investigate decoder-only any-to-any multimodal modeling, which treats all modalities symmetrically and supports arbitrary modalities as inputs and outputs without modality-specific heads, losses, or task pipelines. Because every modality is both an input and an output of the same model, the resulting model, named Modus, can support a range of applications, such as chained generation through intermediate modalities or cross-modal self-verification by scoring the model's own outputs with another generated modality. Modus demonstrates strong out-of-the-box performance and is competitive with specialist and multitask baselines using a single model across various benchmarks. All materials are open-sourced at https://modus-multimodal.epfl.ch/.

多模态解码器仅任意生成开源

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。