arXiv:2511.18262cs.CV2025-11被引 6

统一框架实现文本到图像生成与编辑,性能媲美专用模型

MammothModa2: A Unified AR-Diffusion Framework for Multimodal Understanding and Generation

  • 用自回归规划+扩散生成串联架构,分步完成语义理解与图像合成
  • 在GenEval、DPGBench等测试中得分超0.87、87.2,编辑任务达4.06
  • 无需预训练生成器,端到端训练兼顾生成与理解,适合多模态应用

统一多模态模型致力于在单一框架内整合理解与生成能力,但如何弥合离散语义推理与高保真视觉合成之间的差距仍是挑战。我们提出MammothModa2(Mammoth2),一种统一的自回归-扩散(AR-Diffusion)框架,有效结合自回归语义规划与基于扩散的生成。该框架采用串行设计:自回归路径配备生成专家,在离散标记上进行全局语义建模;单流扩散变换器(DiT)解码器负责高保真图像生成。通过精心设计的AR-Diffusion特征对齐模块,融合多层特征聚合、统一条件编码与上下文条件化,稳定对齐自回归表示与扩散解码器的连续潜在空间。Mammoth2以联合下一词预测与流匹配目标端到端训练,并经监督微调与生成/编辑强化学习优化。仅使用约6000万标注生成样本,不依赖预训练生成器,即可在公开基准上实现文本到图像与指令编辑的优异表现:GenEval得分为0.87,DPGBench为87.2,ImgEdit为4.06,且在多模态理解任务上与Qwen3-VL-8B等仅理解类骨干模型保持竞争力。结果表明,精心耦合的AR-Diffusion架构可在单一、参数与数据高效的模型中同时实现高保真生成与强大多模态理解。

原文摘要 · Abstract (English)

Unified multimodal models aim to integrate understanding and generation within a single framework, yet bridging the gap between discrete semantic reasoning and high-fidelity visual synthesis remains challenging. We present MammothModa2 (Mammoth2), a unified autoregressive-diffusion (AR-Diffusion) framework designed to effectively couple autoregressive semantic planning with diffusion-based generation. Mammoth2 adopts a serial design: an AR path equipped with generation experts performs global semantic modeling over discrete tokens, while a single-stream Diffusion Transformer (DiT) decoder handles high-fidelity image synthesis. A carefully designed AR-Diffusion feature alignment module combines multi-layer feature aggregation, unified condition encoding, and in-context conditioning to stably align AR's representations with the diffusion decoder's continuous latents. Mammoth2 is trained end-to-end with joint Next-Token Prediction and Flow Matching objectives, followed by supervised fine-tuning and reinforcement learning over both generation and editing. With roughly 60M supervised generation samples and no reliance on pre-trained generators, Mammoth2 delivers strong text-to-image and instruction-based editing performance on public benchmarks, achieving 0.87 on GenEval, 87.2 on DPGBench, and 4.06 on ImgEdit, while remaining competitive with understanding-only backbones (e.g., Qwen3-VL-8B) on multimodal understanding tasks. These results suggest that a carefully coupled AR-Diffusion architecture can provide high-fidelity generation and editing while maintaining strong multimodal comprehension within a single, parameter- and data-efficient model.

多模态生成扩散模型自回归图像编辑

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。