让视觉编码器自动调节细粒度到粗粒度的感知，提升多模态模型理解力。
Granulon: Awakening Pixel-Level Visual Encoders with Adaptive Multi-Granularity Semantics for MLLM
- 根据文本语义动态调整视觉抽象层级，实现跨粒度统一推理。
- 在多个基准上提升准确率约30%，幻觉减少约20%。
- 适合需要精细视觉理解的图文任务，如视觉问答与图像描述。
当前多模态大模型主要依赖基于CLIP的视觉编码器，强调全局语义对齐，但在细粒度视觉理解上表现不足。而DINOv3虽具备强像素级感知能力，却缺乏粗粒度语义抽象，限制了多粒度推理。为此，我们提出Granulon，一种基于DINOv3的新型多模态大模型，引入文本条件粒度控制器，根据文本语义范围动态调整视觉抽象层级，并设计自适应令牌聚合模块，通过粒度引导池化与关系感知聚类生成紧凑且语义丰富的视觉令牌。该架构实现单次前向传播内完成“像素→细粒度→粗粒度”的统一推理。大量可解释性实验表明，Granulon在相同设置下准确率提升约30%，幻觉减少约20%，显著优于所有现有视觉编码器。
原文摘要 · Abstract (English)
Recent advances in multimodal large language models largely rely on CLIP-based visual encoders, which emphasize global semantic alignment but struggle with fine-grained visual understanding. In contrast, DINOv3 provides strong pixel-level perception yet lacks coarse-grained semantic abstraction, leading to limited multi-granularity reasoning. To address this gap, we propose Granulon, a novel DINOv3-based MLLM with adaptive granularity augmentation. Granulon introduces a text-conditioned granularity Controller that dynamically adjusts the visual abstraction level according to the semantic scope of the textual input, and an Adaptive Token Aggregation module that performs granularity-guided pooling and relation-aware clustering to produce compact, semantically rich visual tokens. This design enables unified "pixel-to-fine-to-coarse" reasoning within a single forward pass. Extensive and interpretable experiments demonstrate that Granulon improves accuracy by ~30% and reduces hallucination by ~20%, outperforming all visual encoders under identical settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。