统一视觉理解与生成的高效模型,推理速度提升6倍。
LaTtE-Flow: Layerwise Timestep-Expert Flow-based Transformer
- 分层时间专家设计,每层专注特定生成步骤
- 相比同类模型推理快6倍,生成质量相当
- 适合需要实时生成的多模态应用
近期多模态基础模型在统一图像理解和生成方面取得进展,但多数仍需大量预训练,且生成速度慢。本文提出LaTtE-Flow,基于预训练视觉语言模型(VLMs)构建统一框架,引入分层时间专家流模型架构,将流匹配过程分配至不同层组,每组负责特定时间步。该设计仅激活少量层进行采样,显著提升效率。同时提出时间条件残差注意力机制,实现层间信息高效复用。实验表明,该模型在多模态理解任务上表现优异,图像生成质量接近当前最优,推理速度较最新统一模型提升约6倍。
原文摘要 · Abstract (English)
Recent advances in multimodal foundation models unifying image understanding and generation have opened exciting avenues for tackling a wide range of vision-language tasks within a single framework. Despite progress, existing unified models typically require extensive pretraining and struggle to achieve the same level of performance compared to models dedicated to each task. Additionally, many of these models suffer from slow image generation speeds, limiting their practical deployment in real-time or resource-constrained settings. In this work, we propose Layerwise Timestep-Expert Flow-based Transformer (LaTtE-Flow), a novel and efficient architecture that unifies image understanding and generation within a single multimodal model. LaTtE-Flow builds upon powerful pretrained Vision-Language Models (VLMs) to inherit strong multimodal understanding capabilities, and extends them with a novel Layerwise Timestep Experts flow-based architecture for efficient image generation. LaTtE-Flow distributes the flow-matching process across specialized groups of Transformer layers, each responsible for a distinct subset of timesteps. This design significantly improves sampling efficiency by activating only a small subset of layers at each sampling timestep. To further enhance performance, we propose a Timestep-Conditioned Residual Attention mechanism for efficient information reuse across layers. Experiments demonstrate that LaTtE-Flow achieves strong performance on multimodal understanding tasks, while achieving competitive image generation quality with around 6x faster inference speed compared to recent unified multimodal models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。