发现视觉理解与生成在压缩图像令牌时存在本质差异，需协同优化。
On the Limits of Token Reduction for Efficient Unified Vision Language Training
- 针对理解与生成分别设计令牌压缩策略，提升效率
- 联合训练中任务专用压缩导致性能下降15%以上
- 强调保留共享结构对统一模型至关重要，适合高效训练研究者
统一视觉语言模型（VLMs）将视觉理解与生成整合于单一自回归主干网络中，但其联合训练计算成本高昂且缺乏效率考量。本文系统分析层间注意力分配，发现根本性不对称：视觉理解存在显著的深层视觉冗余，而视觉生成则持续依赖图像令牌。基于此，我们设计任务特异性加速器，分别减少各目标的图像令牌计算。虽在独立设置下实现显著效率提升，但在联合训练中却出现一致的协同增益损失——任务专用令牌丢弃导致参数路径分化，消除联合优化中的相互促进效应。结果表明，高效统一建模需保持共享跨任务结构，亟需具备协同意识的加速策略。
原文摘要 · Abstract (English)
Unified vision-language models (VLMs) integrate visual understanding and visual generation within a single autoregressive backbone, but their joint training is computationally expensive and largely overlooked from an efficiency perspective. In this work, we study the feasibility and limits of token-reduction-based acceleration for unified VLM training. Through a systematic analysis of layerwise attention allocation, we uncover a fundamental asymmetry: visual understanding exhibits substantial late-layer visual redundancy, whereas visual generation maintains persistent dependence on image tokens across depth. Guided by this observation, we design task-specific accelerators that selectively reduce image-token computation for each objective. While these methods achieve significant efficiency gains in isolated settings, we observe a consistent synergy loss under unified training -- task-specific token dropping necessitates divergent parameter pathways and eliminates the mutual performance gains typically observed in joint optimization. Our findings suggest that efficient unified modeling requires preserving shared cross-task structures, highlighting the need for synergy-aware acceleration strategies. Project page: https://chicychen.github.io/TokenReductionUnifiedVLM/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。