用字节对编码统一视觉与文本表示，提升多模态理解能力
Unified Multimodal Understanding via Byte-Pair Visual Encoding
- 将字节对编码应用于视觉令牌，融合结构信息
- 在多个视觉语言任务上实现性能提升
- 适合研究多模态大模型与跨模态对齐的学者
多模态大语言模型在视觉-语言理解方面取得显著进展，但有效对齐不同模态仍是根本挑战。我们提出一种通过字节对编码（Byte-Pair Encoding）统一多模态理解的框架。不同于依赖模态专用编码器的传统方法，本方法直接将结构信息嵌入视觉令牌，模仿纯文本语言模型中的成功分词策略。引入基于优先级的编码方案，同时考虑频率与空间一致性，并采用基于课程驱动数据组合的多阶段训练流程。这些改进使Transformer模型更有效地捕捉跨模态关系并利用视觉信息进行推理。全面实验表明，在多种视觉-语言任务中性能均有所提升。该方法弥合了视觉与文本表示之间的差距，推动更强大、高效的多模态基础模型发展。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have made significant progress in vision-language understanding, yet effectively aligning different modalities remains a fundamental challenge. We present a framework that unifies multimodal understanding by applying byte-pair encoding to visual tokens. Unlike conventional approaches that rely on modality-specific encoders, our method directly incorporates structural information into visual tokens, mirroring successful tokenization strategies in text-only language models. We introduce a priority-guided encoding scheme that considers both frequency and spatial consistency, coupled with a multi-stage training procedure based on curriculum-driven data composition. These enhancements enable the transformer model to better capture cross-modal relationships and reason with visual information. Comprehensive experiments demonstrate improved performance across diverse vision-language tasks. By bridging the gap between visual and textual representations, our approach contributes to the advancement of more capable and efficient multimodal foundation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。