轻量级方法提升多模态大模型的图像细节理解能力
HIRE: Lightweight High-Resolution Image Feature Enrichment for Multimodal LLMs
- 用浅层网络实现高分辨率图像特征增强，避免复杂编码器重复调用
- 在多个基准测试中表现接近主流方法，训练推理速度提升显著
- 适合资源受限场景下需要高精度视觉理解的应用
将高分辨率图像特征融入现代多模态大语言模型，显著提升了细粒度视觉理解任务的表现，在多个基准上取得优异成绩。由于这些特征来自大型图像编码器（如ViT），其多次调用带来了显著的计算开销。本文首先提出特征上采样是生成高分辨率特征的自然延伸。通过大量实验与消融分析，证明一个浅层特征增强器可在保持竞争力结果的同时，大幅降低训练与推理时间及计算成本，最多减少1.5倍的浮点运算量（FLOPs）。
原文摘要 · Abstract (English)
The integration of high-resolution image features in modern multimodal large language models has demonstrated significant improvements in fine-grained visual understanding tasks, achieving high performance across multiple benchmarks. Since these features are obtained from large image encoders like ViT, they come with a significant increase in computational costs due to multiple calls to these encoders. In this work, we first develop an intuition for feature upsampling as a natural extension of high-resolution feature generation. Through extensive experiments and ablations, we demonstrate how a shallow feature enricher can achieve competitive results with tremendous reductions in training and inference time as well as computational cost, with upto 1.5x saving in FLOPs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。