arXiv:2507.21741cs.CVcs.MM2025-07IJCAI被引 1

通过语义对齐提升视觉与文本信息融合,增强多模态生成能力。

MAGE: Multimodal Alignment and Generation Enhancement via Bridging Visual and Semantic Spaces

  • 设计智能对齐网络,实现视觉与文本空间的维度和语义对齐。
  • 结合交叉熵与均方误差训练,显著缩小异构数据间的语义差距。
  • 构建指令微调数据集,扩展模型在任意输入输出场景下的能力边界。

在多模态学习的最新进展中,编码后视觉数据的空间与语义损失仍是关键挑战。大模型性能与视觉编码器和大语言模型之间的耦合度正相关。现有方法常因向量间隙或语义差异导致信息丢失。为此,我们提出MAGE(Multimodal Alignment and Generation Enhancement),通过创新的对齐机制桥接视觉与文本的语义空间。引入智能对齐网络(IAN),实现维度与语义对齐;采用交叉熵与均方误差联合训练策略,显著提升对齐效果;为增强MAGE的“任意到任意”能力,构建了多模态工具调用指令微调数据集,拓展输出能力边界。最终,所提多模态大模型在多个评估基准(MME、MMBench、SEED)上表现优于同类工作。完整代码与附录见:https://github.com/GTCOM-NLP/MAGE。

原文摘要 · Abstract (English)

In the latest advancements in multimodal learning, effectively addressing the spatial and semantic losses of visual data after encoding remains a critical challenge. This is because the performance of large multimodal models is positively correlated with the coupling between visual encoders and large language models. Existing approaches often face issues such as vector gaps or semantic disparities, resulting in information loss during the propagation process. To address these issues, we propose MAGE (Multimodal Alignment and Generation Enhancement), a novel framework that bridges the semantic spaces of vision and text through an innovative alignment mechanism. By introducing the Intelligent Alignment Network (IAN), MAGE achieves dimensional and semantic alignment. To reduce the gap between synonymous heterogeneous data, we employ a training strategy that combines cross-entropy and mean squared error, significantly enhancing the alignment effect. Moreover, to enhance MAGE's "Any-to-Any" capability, we developed a fine-tuning dataset for multimodal tool-calling instructions to expand the model's output capability boundaries. Finally, our proposed multimodal large model architecture, MAGE, achieved significantly better performance compared to similar works across various evaluation benchmarks, including MME, MMBench, and SEED. Complete code and appendix are available at: https://github.com/GTCOM-NLP/MAGE.

多模态对齐生成增强大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。