让视觉模型专注关键局部,高效提升细粒度理解能力。
FAVE: Foveated Adaptive Visual Encoding for Efficient Fine-Grained Visual Understanding

- 用可变分辨率设计,仅对重点区域高精度编码
- 在小物体上实现9.4%准确率提升,计算量降低12.7倍
- 适合需要高效处理细节的视觉任务,如图文问答
细粒度视觉理解依赖局部细节,但传统视觉编码器面临全图高分辨率处理代价高与全局压缩编码弱化细节之间的权衡。受人类主动视觉启发,本文将关注位置与编码方式分离,提出轻量级可变分辨率ViT——FAVE,对选定区域以高清晰度编码,同时保持原始几何结构。在图像中对象最大边长为96像素的小物体设定下,通过使用真实标注裁剪区域作为基准,FAVE在相同裁剪窗口内相比固定分辨率ViT提升Top-1准确率9.4个百分点,且计算量降低12.7倍。增加全局分辨率或主干网络容量均无法达到同等性能点。随后,将FAVE作为互补局部分支集成至FastVLM:其局部令牌与原始全局视觉令牌融合,原全局路径和语言模型保持冻结。仅需最多16个额外局部令牌,使TextVQA准确率提升1.60点,并实现3.3倍于SmolVLM2-2.2B的可控首次响应时间(TTFT)加速;在GQA属性问题上,使FastVLM-1.5B提升1.31点,缩小与FastVLM-7B的差距。结果表明,选择性分配高精度局部容量,是高效补充全局表示与模型扩展的细粒度理解新路径。
原文摘要 · Abstract (English)
Fine-grained visual understanding depends on local detail, yet visual encoders face a trade-off between costly full-image high-resolution processing and compact global encoding that can weaken such evidence. Inspired by human active vision, we separate where to look from what to encode. We focus on the latter and introduce FAVE (Foveated Adaptive Visual Encoding), a lightweight variable-resolution ViT that encodes externally selected regions at high acuity while preserving native geometry. We first isolate this encoding problem using oracle ground-truth crops in a controlled small-object regime. On ImageNet objects with a native maximum side of 96 pixels, FAVE improves Top-1 by 9.4 points over a fixed-resolution ViT on the same crop window with 12.7 times lower FLOPs. Increasing global resolution or backbone capacity does not recover the same operating point. We then integrate FAVE as a complementary local branch in FastVLM. Its local tokens are combined with FastVLM's global visual tokens, while the original global pathway and language model remain frozen. With at most 16 additional local tokens, FAVE improves TextVQA by 1.60 points and achieves a 3.3 times controlled TTFT speedup over SmolVLM2-2.2B. On GQA attribute questions, it improves FastVLM-1.5B by 1.31 points, extending the benefit beyond text while narrowing the gap to FastVLM-7B. Together, these results show that selectively allocating high-acuity local capacity provides an efficient complement to broader global representations and model scaling for fine-grained understanding of small objects, text, and attributes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。