arXiv:2603.21701cs.CVcs.AI2026-03

提出新方法,让大模型多轮对话更高效不丢信息

Rethinking Token Reduction for Large Vision-Language Models

  • 用可学习压缩映射统一剪枝与合并,摆脱人工设计
  • 在多轮视觉问答中实现更高效率与准确率平衡
  • 适合追求推理速度又不想损失多轮理解的开发者

大型视觉语言模型在视觉理解与推理方面表现优异,但过多的视觉标记导致推理成本过高。尽管近期有方法缓解此问题,但主要针对单轮视觉问答(VQA),对更实用的多轮视觉问答(MT-VQA)关注不足。多轮场景下后续问题未知且可能指向任意图像区域,现有压缩策略失效。当前方法分为两类:依赖提示的方法偏向初始文本,忽略后续对话所需信息;无提示方法虽可适用,但依赖注意力分数等启发式指标,性能不佳。本文提出一种基于学习的无提示方法MetaCompress,将标记压缩建模为可学习的压缩映射,统一剪枝与合并为单一学习目标。在此基础上,设计数据高效训练范式,在有限计算开销下学习最优压缩策略。在多个多轮视觉问答基准及多种LVLM架构上实验表明,MetaCompress在效率-精度权衡上优于现有方法,并保持良好的跨对话轮次泛化能力。代码已公开于https://github.com/MArSha1147/MetaCompress。

原文摘要 · Abstract (English)

Large Vision-Language Models (LVLMs) excel in visual understanding and reasoning, but the excessive visual tokens lead to high inference costs. Although recent token reduction methods mitigate this issue, they mainly target single-turn Visual Question Answering (VQA), leaving the more practical multi-turn VQA (MT-VQA) scenario largely unexplored. MT-VQA introduces additional challenges, as subsequent questions are unknown beforehand and may refer to arbitrary image regions, making existing reduction strategies ineffective. Specifically, current approaches fall into two categories: prompt-dependent methods, which bias toward the initial text prompt and discard information useful for subsequent turns; prompt-agnostic ones, which, though technically applicable to multi-turn settings, rely on heuristic reduction metrics such as attention scores, leading to suboptimal performance. In this paper, we propose a learning-based prompt-agnostic method, termed MetaCompress, overcoming the limitations of heuristic designs. We begin by formulating token reduction as a learnable compression mapping, unifying existing formats such as pruning and merging into a single learning objective. Upon this formulation, we introduce a data-efficient training paradigm capable of learning optimal compression mappings with limited computational costs. Extensive experiments on MT-VQA benchmarks and across multiple LVLM architectures demonstrate that MetaCompress achieves superior efficiency-accuracy trade-offs while maintaining strong generalization across dialogue turns. Our code is available at https://github.com/MArSha1147/MetaCompress.

视觉语言模型多轮对话高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。