统一输入编码,让多模态模型不再因模态差异而耗能。
Mind the Gap No More: Achieving Zero-Gap Multimodal Integration via One Tokenizer
- 所有模态直接映射到共享词元空间,消除跨模态间隙。
- 在DNA-文本任务中性能超越传统模块化架构,推理更高效。
- 适合需要深度跨模态理解的生物信息学研究者。
多模态大语言模型(MLLM)的核心挑战在于如何将异构输入整合为连贯的推理引擎。当前方法普遍采用模块化设计,依赖模态特异性编码器和跨模态融合机制,但这类结构受制于几何模态间隙,迫使大语言模型大量计算资源用于几何对齐而非深层跨模态推理。本文首次形式化定义该模态间隙,并理论证明:采用统一词表的原生架构能在所有隐藏层保持零间隙状态。基于此,我们提出「One Tokenizer」——一种将所有模态直接映射至共享词元空间的原生架构。在DNA-文本多模态测试平台上的实证验证表明,通过在模型原生潜在空间实现无缝集成,One Tokenizer持续优于基于编码器的模块化模型,为深度生物推理提供更优框架。
原文摘要 · Abstract (English)
A central challenge in developing Multimodal Large Language Models (MLLMs) is effectively integrating heterogeneous inputs into a cohesive reasoning engine. Current paradigms predominantly rely on modular architectures that introduce modality-specific encoders and cross-modal fusion mechanisms. However, these designs are fundamentally bottlenecked by a geometric modality gap, forcing the LLM to expend significant computational capacity on geometric reconciliation rather than deep cross-modal reasoning. In this work, we formally characterize this modality gap and theoretically demonstrate that native architectures, specifically those employing a unified vocabulary, intrinsically maintain a zero-gap state across all hidden layers. Guided by these theoretical findings, we propose \textit{One Tokenizer}, a native architecture that maps all modalities directly into a shared token space. We empirically validate this framework on a DNA--text multimodal testbed. Our extensive evaluations reveal that by achieving seamless integration within the LLM's native latent space, One Tokenizer consistently outperforms encoder-based modular counterparts, providing a fundamentally superior framework for deep biological reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。