arXiv:2605.12309cs.CV2026-05被引 2

提出新方法减少视觉标记,提升多模态模型效率与编辑能力。

G$^2$TR: Generation-Guided Visual Token Reduction for Separate-Encoder Unified Multimodal Models

论文配图:G$^2$TR: Generation-Guided Visual Token Reduction for Separate-Encoder Unified Multimodal Models
图 1 · 摘自论文原文
  • 利用生成分支信号判断视觉标记重要性,兼顾语义与重建。
  • 减少1.94倍视觉标记和预填充计算,保持推理与编辑质量。
  • 无需训练、即插即用,适合已有多模态模型部署优化。

分离编码器统一多模态模型(UMMs)因密集视觉标记处理导致推理成本快速上升。本文聚焦理解阶段的视觉标记压缩以提升效率。现有方法多依赖注意力分数或图文相似度,隐含假设最终目标为判别性推理,但此假设不适用于需支持图像编辑的UMMs。为此,提出G$^2$TR:一种生成引导的视觉标记压缩框架。其核心洞察是生成分支提供任务无关信号,可识别既语义相关又对潜在空间图像重建至关重要的理解型视觉标记。G$^2$TR通过与VAE潜变量的一致性估计标记重要性,进行平衡选择并合并冗余标记,降低信息损失。该方法无需训练、即插即用,仅在理解编码后应用,兼容现有推理流程。在图像理解与编辑基准测试中,G$^2$TR将视觉标记和预填充计算减少1.94倍,同时维持推理准确率与编辑质量,在几乎所有基准上优于基线。代码已开源。

原文摘要 · Abstract (English)

The development of separate-encoder Unified multimodal models (UMMs) comes with a rapidly growing inference cost due to dense visual token processing. In this paper, we focus on understanding-side visual token reduction for improving the efficiency of separate-encoder UMMs. While this topic has been widely studied for MLLMs, existing methods typically rely on attention scores, text-image similarity and so on, implicitly assuming that the final objective is discriminative reasoning. This assumption does not hold for UMMs, where understanding-side visual tokens must also preserve the model's capabilities for editing images. We propose G$^2$TR, a generation-guided visual token reduction framework for separate-encoder UMMs. Our key insight is that the generation branch provides a task-agnostic signal for identifying understanding-side visual tokens that are not only semantically relevant but also important for latent-space image reconstruction and generation. G$^2$TR estimates token importance from consistency with VAE latent, performs balanced token selection, and merges redundant tokens into retained representatives to reduce information loss. The method is training-free, plug-and-play, and applied only after the understanding encoding stage, making it compatible with existing UMM inference pipelines. Experiments on image understanding and editing benchmarks show that G$^2$TR substantially reduces visual tokens and prefill computation by 1.94x while maintaining both reasoning accuracy and editing quality, outperforming baselines on almost all benchmarks. Code is at: https://github.com/lijunxian111/G2TR.

多模态视觉压缩生成引导高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。