arXiv:2503.23959cs.CV2025-03被引 2

通过保留局部视觉细节,实现图像对话生成的高效加速。

Local Information Matters: Inference Acceleration For Grounded Conversation Generation Models Through Adaptive Local-Aware Token Pruning

  • 基于超像素分割与动态密度分配,优先保留关键物体区域的视觉令牌。
  • 在GranDf数据集上,令牌减少90%时,AP50提升4.9%,召回率提升5.0%。
  • 适合需要高精度视觉定位的多模态对话系统优化,尤其关注计算效率。

基于视觉的语言对话生成(GCG)是一项新兴任务,要求模型生成与物体分割掩码紧密结合的自然语言响应。近期模型如GLaMM和OMG-LLaVA虽实现像素级定位,但因处理大量视觉令牌而带来显著计算开销。现有剪枝方法如FastV和PyramidDrop未能保留对定位至关重要的局部特征,导致性能大幅下降。为此,我们提出自适应局部感知令牌剪枝(ALTP),通过两个核心组件:(1)细节密度捕捉(DDC),利用超像素分割保留以物体为中心区域的令牌,保持细粒度信息;(2)动态密度形成(DDF),根据信息密度动态分配令牌,确保语义丰富区域更高保留率。在GranDf数据集上的大量实验表明,ALTP显著优于现有剪枝方法。应用于GLaMM时,令牌减少90%的同时,AP50提升4.9%,召回率提升5.0%,优于PyramidDrop;在OMG-LLaVA上,令牌减少90%时,AP提升2.1%,mIOU提升3.0%,优于PDrop。

原文摘要 · Abstract (English)

Grounded Conversation Generation (GCG) is an emerging vision-language task that requires models to generate natural language responses seamlessly intertwined with corresponding object segmentation masks. Recent models, such as GLaMM and OMG-LLaVA, achieve pixel-level grounding but incur significant computational costs due to processing a large number of visual tokens. Existing token pruning methods, like FastV and PyramidDrop, fail to preserve the local visual features critical for accurate grounding, leading to substantial performance drops in GCG tasks. To address this, we propose Adaptive Local-Aware Token Pruning (ALTP), a simple yet effective framework that accelerates GCG models by prioritizing local object information. ALTP introduces two key components: (1) Detail Density Capture (DDC), which uses superpixel segmentation to retain tokens in object-centric regions, preserving fine-grained details, and (2) Dynamic Density Formation (DDF), which dynamically allocates tokens based on information density, ensuring higher retention in semantically rich areas. Extensive experiments on the GranDf dataset demonstrate that ALTP significantly outperforms existing token pruning methods, such as FastV and PyramidDrop, on both GLaMM and OMG-LLaVA models. Notably, when applied to GLaMM, ALTP achieves a 90% reduction in visual tokens with a 4.9% improvement in AP50 and a 5.0% improvement in Recall compared to PyramidDrop. Similarly, on OMG-LLaVA, ALTP improves AP by 2.1% and mIOU by 3.0% at a 90% token reduction compared with PDrop.

视觉对话令牌剪枝多模态加速

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。