轻量级双融合框架，用现成模型高效实现图文统一理解与生成
LightFusion: A Light-weighted, Double Fusion Framework for Unified Multimodal Understanding and Generation
- 保留原始模型结构，插入多模态自注意力块实现双路融合
- 仅用350亿词训练即在多个评测中达到领先水平
- 适合想快速构建高效多模态系统的研究者和开发者
统一多模态模型近年来在能力与通用性上取得显著进展，但多数主流系统仍需从头训练，消耗大量计算资源。本文表明，通过战略性地融合专精于生成或理解的公开模型，可更高效地获得竞争力表现。核心设计是保留原有模块，并在网络中交错插入多模态自注意力块，形成双融合机制：(1) 实现丰富的多模态融合，同时最大程度保留基模型原有优势；(2) 促进理解编码器的高层语义表示与生成编码器的低层空间信号协同融合。仅使用约350亿词训练，该方法在多个基准测试中表现优异：在GenEval上达0.91分（组合文本到图像生成），DPG-Bench上82.16分（复杂文本到图像生成），GEditBench上6.06分，ImgEdit-Bench上3.77分（图像编辑）。我们已完全开放代码、模型权重及数据集，以支持未来统一多模态建模研究。
原文摘要 · Abstract (English)
Unified multimodal models have recently shown remarkable gains in both capability and versatility, yet most leading systems are still trained from scratch and require substantial computational resources. In this paper, we show that competitive performance can be obtained far more efficiently by strategically fusing publicly available models specialized for either generation or understanding. Our key design is to retain the original blocks while additionally interleaving multimodal self-attention blocks throughout the networks. This double fusion mechanism (1) effectively enables rich multi-modal fusion while largely preserving the original strengths of the base models, and (2) catalyzes synergistic fusion of high-level semantic representations from the understanding encoder with low-level spatial signals from the generation encoder. By training with only ~ 35B tokens, this approach achieves strong results across multiple benchmarks: 0.91 on GenEval for compositional text-to-image generation, 82.16 on DPG-Bench for complex text-to-image generation, 6.06 on GEditBench, and 3.77 on ImgEdit-Bench for image editing. By fully releasing the entire suite of code, model weights, and datasets, we hope to support future research on unified multimodal modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。