提出X形结构解决多模态模型训练时视觉与文本的梯度冲突问题。
Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Models
- 采用两头分治、中间共享的X形架构，分离视觉与文本处理层。
- 在3B参数规模下达到7B模型的生成性能，图像生成得分为82。
- 适合追求高效训练和可扩展性的多模态系统研发者使用。
基于共享自回归（AR）Transformer的统一多模态模型（UMMs）因其架构简洁而备受关注。然而，我们发现其核心缺陷：在多模态输入训练时，视觉与文本间存在严重的梯度冲突，尤其在浅层和深层。根源在于图像与文本的底层统计特性差异显著，而中层表示更抽象、语义对齐后冲突减弱。为此，我们提出Uni-X，一种两头分离、中间共享的X形架构。该设计将初始与末端层专用于模态特定处理，中间层保持共享以实现高层语义融合。此结构不仅消除两端梯度冲突，还进一步缓解共享层中的残余冲突。大量实验验证其有效性：相同训练条件下，Uni-X训练效率更优；扩展至3B参数并使用更大数据集后，其性能媲美甚至超越7B AR-based UMMs，图像生成任务的GenEval得分为82，同时在文本与视觉理解任务上表现优异。结果表明，Uni-X是未来统一多模态建模中参数高效且可扩展的基石。代码已开源于https://github.com/CURRENTF/Uni-X。
原文摘要 · Abstract (English)
Unified Multimodal Models (UMMs) built on shared autoregressive (AR) transformers are attractive for their architectural simplicity. However, we identify a critical limitation: when trained on multimodal inputs, modality-shared transformers suffer from severe gradient conflicts between vision and text, particularly in shallow and deep layers. We trace this issue to the fundamentally different low-level statistical properties of images and text, while noting that conflicts diminish in middle layers where representations become more abstract and semantically aligned. To overcome this challenge, we propose Uni-X, a two-end-separated, middle-shared architecture. Uni-X dedicates its initial and final layers to modality-specific processing, while maintaining shared parameters in the middle layers for high-level semantic fusion. This X-shaped design not only eliminates gradient conflicts at both ends but also further alleviates residual conflicts in the shared layers. Extensive experiments validate the effectiveness of Uni-X. Under identical training conditions, Uni-X achieves superior training efficiency compared to strong baselines. When scaled to 3B parameters with larger training data, Uni-X matches or surpasses 7B AR-based UMMs, achieving a GenEval score of 82 for image generation alongside strong performance in text and vision understanding tasks. These results establish Uni-X as a parameter-efficient and scalable foundation for future unified multimodal modeling. Our code is available at https://github.com/CURRENTF/Uni-X
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。