通过分层并行交互提升多模态生成质量
ParaUni: Enhance Generation in Unified Multimodal Model with Reinforcement-driven Hierarchical Parallel Information Interaction
- 从视觉语言模型多层提取特征并行融合
- 多层特征融合后生成质量显著提升
- 适合追求高质量图像生成的研究者
统一多模态模型通过结合视觉语言模型(VLM)与扩散模型显著提升了视觉生成能力。然而,现有方法因表征差异大,难以同时实现充分交互与灵活部署。鉴于VLM各层包含从低级细节到高级语义的丰富层次信息,本文提出ParaUni:以并行方式提取VLM多层特征,实现全面信息交互,并保持灵活分离结构以增强统一多模态模型的生成性能。具体地,所有VLM层的视觉特征并行输入层融合模块(LIM),高效整合细粒度细节与语义抽象,将融合表示作为条件输入扩散模型。为进一步提升性能,我们发现不同层级对强化学习(RL)中各类奖励响应不均。因此设计层间动态调整机制(LDAM),利用RL对齐各层级的层次特性,实现多重奖励优化。大量实验表明,ParaUni有效利用互补多层特征,显著提升生成质量,并在RL阶段展现多重奖励优化潜力。代码已开源。
原文摘要 · Abstract (English)
Unified multimodal models significantly improve visual generation by combining vision-language models (VLMs) with diffusion models. However, existing methods struggle to fully balance sufficient interaction and flexible implementation due to vast representation difference. Considering abundant and hierarchical information in VLM's layers from low-level details to high-level semantics, we propose \textbf{ParaUni}. It extracts features from variants VLM's layers in a \textbf{Para}llel way for comprehensive information interaction and retains a flexible separation architecture to enhance generation in \textbf{Uni}fied multimodal model. Concretely, visual features from all VLM's layers are fed in parallel into a Layer Integration Module (LIM), which efficiently integrates fine-grained details and semantic abstractions and provides the fused representation as a condition to the diffusion model. To further enhance performance, we reveal that these hierarchical layers respond unequally to different rewards in Reinforcement Learning (RL). Crucially, we design a Layer-wise Dynamic Adjustment Mechanism (LDAM) to facilitate multiple reward improvements that aligns the hierarchical properties of these layers using RL. Extensive experiments show ParaUni leverages complementary multi-layer features to substantially improve generation quality and shows strong potential for multiple reward advances during RL stages. Code is available at https://github.com/JosephTiTan/ParaUni.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。