解决视觉语言模型的分辨率难题,提升原生图像处理能力。
Native Visual Understanding: Resolving Resolution Dilemmas in Vision-Language Models
- 提出原生分辨率编码框架,支持图像原始尺寸输入
- 在极端分辨率下性能显著优于传统低分辨率模型
- 开源工具链适合研究多尺度视觉理解的开发者
视觉语言模型(VLMs)在处理真实世界图像的多样化分辨率和宽高比时面临严峻挑战,因多数现有模型依赖固定且低分辨率的输入。尽管近期研究探索了原生分辨率视觉编码以提升性能,但这些工作仍零散且缺乏开源社区中的系统性框架。此外,现有基准测试在评估VLMs时未能充分考虑分辨率这一关键因素。为应对由模型设计与评测局限引发的“分辨率困境”,我们提出了RC-Bench——一个专门针对极端视觉条件系统评估VLM能力的新基准,重点关注分辨率与宽高比变化。同时,我们推出了NativeRes-LLaVA,一个开源训练框架,使VLM能有效处理图像的原始分辨率与宽高比。基于此,我们在现有视觉编码策略上开展全面实验,结果表明:原生分辨率视觉编码显著提升了VLM在RC-Bench及其他聚焦分辨率的基准上的表现。代码已开源:https://github.com/Niujunbo2002/NativeRes-LLaVA。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) face significant challenges when dealing with the diverse resolutions and aspect ratios of real-world images, as most existing models rely on fixed, low-resolution inputs. While recent studies have explored integrating native resolution visual encoding to improve model performance, such efforts remain fragmented and lack a systematic framework within the open-source community. Moreover, existing benchmarks fall short in evaluating VLMs under varied visual conditions, often neglecting resolution as a critical factor. To address the "Resolution Dilemma" stemming from both model design and benchmark limitations, we introduce RC-Bench, a novel benchmark specifically designed to systematically evaluate VLM capabilities under extreme visual conditions, with an emphasis on resolution and aspect ratio variations. In conjunction, we propose NativeRes-LLaVA, an open-source training framework that empowers VLMs to effectively process images at their native resolutions and aspect ratios. Based on RC-Bench and NativeRes-LLaVA, we conduct comprehensive experiments on existing visual encoding strategies. The results show that Native Resolution Visual Encoding significantly improves the performance of VLMs on RC-Bench as well as other resolution-centric benchmarks. Code is available at https://github.com/Niujunbo2002/NativeRes-LLaVA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。