用双曲空间提升多模态大模型训练效率，让视觉与文本对齐更精准。
HyperET: Efficient Training in Hyperbolic Space for Multi-modal Large Language Models
- 在双曲空间中动态调整半径，实现跨模态任意粒度对齐。
- 仅增加不到1%参数，显著提升多模态模型预训练与微调性能。
- 适合追求高效训练的多模态大模型研究者和开发者。
多模态大语言模型（MLLMs）已成为实现视觉与文本理解对齐的变革性方法，但通常需要数千个GPU进行训练以实现多粒度级别的跨模态对齐。本文认为，这种低效的关键原因在于广泛使用的视觉编码器（如CLIP和SAM）缺乏与语言在多粒度层级上的对齐。为此，本文提出一种基于双曲空间的高效训练范式HyperET，利用双曲空间天然建模层次结构的特性，在任意粒度层级上通过动态调整双曲半径优化视觉表征，使其与文本对应项对齐。HyperET采用可学习矩阵结合莫比乌斯乘法运算，提供三种有效配置：对角缩放矩阵、分块对角矩阵和带状矩阵，实现了灵活且高效的参数化策略。在多个MLLM基准测试中，实验结果表明，HyperET能持续提升现有预训练与微调模型性能，且额外参数不足1%。代码已开源：https://github.com/godlin-sjtu/HyperET。
原文摘要 · Abstract (English)
Multi-modal large language models (MLLMs) have emerged as a transformative approach for aligning visual and textual understanding. They typically require extremely high computational resources (e.g., thousands of GPUs) for training to achieve cross-modal alignment at multi-granularity levels. We argue that a key source of this inefficiency lies in the vision encoders they widely equip with, e.g., CLIP and SAM, which lack the alignment with language at multi-granularity levels. To address this issue, in this paper, we leverage hyperbolic space, which inherently models hierarchical levels and thus provides a principled framework for bridging the granularity gap between visual and textual modalities at an arbitrary granularity level. Concretely, we propose an efficient training paradigm for MLLMs, dubbed as HyperET, which can optimize visual representations to align with their textual counterparts at an arbitrary granularity level through dynamic hyperbolic radius adjustment in hyperbolic space. HyperET employs learnable matrices with Möbius multiplication operations, implemented via three effective configurations: diagonal scaling matrices, block-diagonal matrices, and banded matrices, providing a flexible yet efficient parametrization strategy. Comprehensive experiments across multiple MLLM benchmarks demonstrate that HyperET consistently improves both existing pre-training and fine-tuning MLLMs clearly with less than 1\% additional parameters. Code is available at https://github.com/godlin-sjtu/HyperET
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。