提出无需训练的分层解耦框架,提升高分辨率图像理解性能。
HiDe: Rethinking The Zoom-IN method in High Resolution MLLMs via Hierarchical Decoupling
- 通过分层解耦分离问题与视觉关键区域,精准定位目标信息。
- 在V*Bench等数据集上达92.1%准确率,超越现有SOTA及强化学习方法。
- 内存消耗减少75%,适合资源受限场景下的高效视觉推理。
多模态大语言模型(MLLMs)在视觉理解任务中取得显著进展,但在高分辨率图像上的表现仍不理想。现有方法常归因于感知限制和小物体识别困难,采用“缩放”策略以获取更细粒度信息。然而,我们的分析表明,主要瓶颈并非物体尺寸,而是复杂背景干扰。通过一系列解耦实验,我们提出无需训练的分层解耦框架(HiDe),包含逐标记注意力解耦(TAD)和布局保持解耦(LPD)。TAD识别关键视觉区域,利用注意力权重实现精准对齐;LPD将目标区域从背景中解耦,重构保留空间布局的紧凑表示,消除背景干扰。HiDe在V*Bench、HRBench4K和HRBench8K上达到新SOTA,使Qwen2.5-VL 7B和InternVL3 8B分别达到92.1%和91.6%的准确率,超越强化学习方法。优化后,内存使用量仅为前代方法的25%。代码已公开。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have made significant strides in visual understanding tasks. However, their performance on high-resolution images remains suboptimal. While existing approaches often attribute this limitation to perceptual constraints and argue that MLLMs struggle to recognize small objects, leading them to use "zoom in" strategies for better detail, our analysis reveals a different cause: the main issue is not object size, but rather caused by complex background interference. We systematically analyze this "zoom in" operation through a series of decoupling experiments and propose the Hierarchical Decoupling Framework (HiDe), a training-free framework that uses Token-wise Attention Decoupling (TAD) to decouple the question tokens and identify the key information tokens, then leverages their attention weights to achieve precise alignment with the target visual regions. Subsequently, it employs Layout-Preserving Decoupling (LPD) to decouple these regions from the background and reconstructs a compact representation that preserves essential spatial layouts while eliminating background interference. HiDe sets a new SOTA on V*Bench, HRBench4K, and HRBench8K, boosting Qwen2.5-VL 7B and InternVL3 8B to SOTA (92.1% and 91.6% on V*Bench), even surpassing RL methods. After optimization, HiDe uses 75% less memory than the previous training-free approach. Code is provided in https://tennine2077.github.io/HiDe.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。