arXiv:2511.20520cs.CV2025-11被引 9

提出H型异构专家架构,提升多模态理解与生成效果

HBridge: H-Shape Bridging of Heterogeneous Experts for Unified Multimodal Understanding and Generation

  • 采用非对称H形结构,分层选择性连接理解与生成专家
  • 减少40%以上注意力共享,生成质量显著提升
  • 适合需要高效多模态生成的场景,如图文生成

近期统一模型将理解专家(如LLMs)与生成专家(如扩散模型)结合,实现了强大的多模态性能。然而,诸如BAGEL和LMFusion等先进方法遵循Mixture-of-Transformers(MoT)范式,采用对称设计,虽便于初始化与融合,但因模态本质差异仍存在不足。本文提出HBridge,一种非对称的H-shaped架构,使异构专家能最优利用各自模态领域的预训练先验。不同于以往通过共享注意力全层连接专家的密集融合策略,HBridge仅选择性桥接中间层,减少超过40%的注意力共享,提升效率并增强生成质量。浅层与深层分别捕获模态特异性表征,实现解耦;中层桥接促进语义对齐。为进一步强化跨模态一致性,引入语义重建令牌,显式引导生成专家重构目标图像的视觉语义令牌。在多个基准上的实验表明HBridge有效且性能更优,确立了统一多模态生成的新范式。

原文摘要 · Abstract (English)

Recent unified models integrate understanding experts (e.g., LLMs) with generative experts (e.g., diffusion models), achieving strong multimodal performance. However, recent advanced methods such as BAGEL and LMFusion follow the Mixture-of-Transformers (MoT) paradigm, adopting a symmetric design that mirrors one expert to another for convenient initialization and fusion, which remains suboptimal due to inherent modality discrepancies. In this work, we propose HBridge, an asymmetric H-shaped architecture that enables heterogeneous experts to optimally leverage pretrained priors from their respective modality domains. Unlike prior dense fusion strategies that straightforwardly connect all layers between experts via shared attention, HBridge selectively bridges intermediate layers, reducing over 40% attention sharing, which improves efficiency and enhances generation quality. Shallow and deep layers, which capture modality-specific representations, are decoupled, while mid-layer bridging promotes semantic alignment. To further strengthen cross-modal coherence, we introduce semantic reconstruction tokens that explicitly guide the generative expert to reconstruct visual semantic tokens of the target image. Extensive experiments across multiple benchmarks demonstrate the effectiveness and superior performance of HBridge, establishing a new paradigm for unified multimodal generation.

多模态生成扩散模型异构专家架构设计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。