用低分辨率图实现高精度视觉感知，节省算力还更准。
SigLIP-HD by Fine-to-Coarse Supervision
- 通过粗粒度特征模仿细粒度特征，提升模型感知能力
- 在相同计算成本下，视觉令牌质量优于基线模型
- 特别适合需要精细图像理解的场景，如文字识别
高质量视觉表征是计算机视觉长期追求的目标。在多模态大模型（MLLM）中，输入更高分辨率图像可生成更细粒度的视觉标记，但会带来额外计算开销与设计复杂性，需多次前向传播并处理更多标记。在直接采用更高分辨率前，我们是否已充分挖掘标准分辨率下的模型感知潜力？为此，本文提出SigLIP-HD，核心是一种简单高效的从细到粗监督机制：强制中等分辨率图像的粗粒度特征模仿其高分辨率版本的细粒度特征。该框架基于先进的SigLIP 2模型构建，最终在相同推理预算下生成更优视觉标记。在广泛多模态大模型基准测试中，其性能持续优于基线模型，尤其在OCR相关任务上表现突出。
原文摘要 · Abstract (English)
High-quality visual representation is a long-standing pursuit in computer vision. In the context of multimodal LLMs (MLLMs), feeding higher-resolution images can produce more fine-grained visual tokens. However, it introduces additional computational and design complexity, due to multiple forward passes and post-processing of increased tokens. Before simply adopting a higher resolution, have we truly unlocked the model's full perception capability at a standard resolution? Therefore, we study an interesting problem: how to achieve fine visual perception under lower cost without larger images. We present SigLIP-HD in this work. The core is a highly simple fine-to-coarse supervision design. We enforce the coarse feature of a mid-resolution image to mimic the fine-grained feature of its high-resolution version. We build this framework on the advanced SigLIP 2 model. Our final model produces better visual tokens at exactly the same inference budget. It is validated on extensive MLLM benchmarks and consistently delivers stronger results than our baseline model, especially on OCR-related tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。