无需标注数据,用模型自蒸馏实现高效精准的视觉细节定位。
Catching the Details: Self-Distilled RoI Predictors for Fine-Grained MLLM Perception
- 利用模型中间层注意力图自动生成伪区域标签,通过去噪与消歧提升质量。
- 仅需10,000个问答对训练,推理时单次前向传播完成定位,效率显著提升。
- 适用于多种大模型,可直接集成,无需全模型微调,适合实际部署。
多模态大语言模型(MLLM)需要高分辨率视觉信息进行细粒度感知,但处理整幅高分辨率图像计算成本过高。现有基于感兴趣区域(RoI)的方法存在两难:依赖大规模标注数据的训练方法成本高,而无需训练的内部注意力方法计算低效且精度差,常需多轮预填充或依赖缓慢的自回归解码。本文提出一种高效、无标注的自蒸馏区域提议网络(SD-RPN),通过将MLLM中间层的噪声注意力图转化为高质量伪RoI标签,显式去噪并解决模糊性。使用这些标签训练轻量级区域提议网络(RPN),学习更精确的定位能力。该RPN仅需一次前向传播,利用中间层特征即可完成RoI预测,解耦于自回归生成过程,避免耗时的多轮操作。在多个MLLM家族中验证,即使仅用约10,000个问答对训练,仍展现卓越的数据效率与泛化能力,在TextVQA、DocVQA和V-Star等未见基准上实现超过10%的绝对准确率提升。本工作为提升MLLM细粒度感知提供了一种低成本、可扩展的实用方案。代码已开源:https://github.com/YuHengsss/SD-RPN。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) require high-resolution visual information to perform fine-grained perception, yet processing entire high-resolution images is computationally prohibitive. While recent methods leverage a Region-of-Interest (RoI) mechanism to focus on salient areas, they typically present a difficult trade-off: training-based approaches depend on large-scale annotated datasets, while training-free methods that utilize the model's internal attention are computationally inefficient and less accurate, requiring either multi-pass prefill stages or reliance on the slow auto-regressive decoding process. In this paper, we propose an efficient, annotation-free Self-Distilled Region Proposal Network (SD-RPN) that resolves this trade-off. The SD-RPN is built around a pipeline that transforms the noisy attention maps from the MLLM's middle layers into high-quality pseudo-RoI labels by explicitly denoising the signal and resolving ambiguity. We use these labels to train a lightweight Region Proposal Network (RPN) that learns a more precise localization. This RPN is also highly efficient, predicting the RoI in a single forward pass using features from the MLLM's middle layers, decoupling RoI identification from the auto-regressive generation and avoiding costly multi-pass operations. To validate our approach, we integrate the framework into multiple MLLM families. Despite being trained on only a few (e.g. 10K) question-answer pairs, our method demonstrates exceptional data efficiency and generalization, achieving over a 10% absolute accuracy improvement on unseen benchmarks, including TextVQA, DocVQA, and V-Star. Our work presents a practical and scalable solution for enhancing the fine-grained perception of MLLMs without requiring costly supervision or full model fine-tuning. Code is available at https://github.com/YuHengsss/SD-RPN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。