让大模型一次生成图文音视频多种内容，突破传统单对单模态限制。
Spider: Any-to-Many Multimodal LLM
- 设计新框架Spider，支持任意组合多模态输出
- 构建首个文本格式的多模态数据集TMM，训练模型生成多模态内容
- 可生成含图像、音频、视频的复合输出，适合多模态应用开发者
多模态大模型（MLLM）扩展了语言模型的能力，但现有Any-to-Any MLLM仅能在单一响应中生成一对模态，如文本+图像或文本+音频。为突破此限制，本文提出Spider——一种高效的任意对多模态生成（AMMG）框架，可同时生成任意组合的多模态内容，如文本+图像+音频+视频。Spider融合三大组件：基础模型处理任意模态转换，定制化指令模板生成多模态信号提示，以及高效解码器控制器协调多个解码器生成多模态内容。为训练该框架，我们构建了全新的文本格式多模态数据集TMM，支持学习任意对多模态能力。最终，训练后的Spider生成首个伪X-to-Xs多模态数据集，为未来研究提供重要数据支撑。本工作不仅拓展了多模态交互边界，也为领域发展提供了丰富数据资源。
原文摘要 · Abstract (English)
Multimodal LLMs (MLLMs) have emerged as an extension of Large Language Models (LLMs), enabling the integration of various modalities. However, Any-to-Any MLLMs are limited to generating pairwise modalities 'Text + X' within a single response, such as Text + {Image or Audio or Video}. To address this limitation, we introduce Spider, a novel efficient Any-to-Many Modalities Generation (AMMG) framework, which can generate an arbitrary combination of modalities 'Text + Xs', such as Text + {Image and Audio and Video}. To achieve efficient AMMG, our Spider integrates three core components: a Base Model for basic X-to-X (i.e., Any-to-Any) modality processing, an Any-to-Many Instruction Template designed for producing Xs signal prompts, and a novel Efficient Decoders-Controller for controlling multimodal Decoders to generate Xs (many-modal) contents. To train Spider, we constructed a novel Text-formatted Many-Modal (TMM) dataset, which facilitates learning the X-to-Xs (i.e., Any-to-Many) capability necessary for AMMG. Ultimately, the well-trained Spider generates a pseudo X-to-Xs dataset, the first-ever X-to-Xs many-modal dataset, enhancing the potential for AMMG tasks in future research. Overall, this work not only pushes the boundary of multimodal interaction but also provides rich data support for advancing the field. Code: https://github.com/Layjins/Spider
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。