arXiv:2512.18496cs.CV2025-12被引 1

根据图像复杂度动态压缩视觉特征,提升多模态模型效率与性能

Adaptive-VoCo: Complexity-Aware Visual Token Compression for Vision-Language Models

  • 基于视觉编码器的统计特征动态判断图像复杂度,自适应调整压缩率
  • 在多个任务上优于固定压缩率方法,保持高精度同时降低计算开销
  • 适合追求高效推理的多模态应用,如移动端视觉理解

近年来,大规模视觉语言模型(VLMs)在多模态理解与推理任务中表现出色。然而,处理高维视觉特征常带来巨大计算与内存开销。VoCo-LLaMA通过将视觉补丁令牌压缩为少量VoCo令牌,降低了计算负担并保持跨模态对齐。但此类方法通常采用固定压缩率,难以适应不同图像的视觉复杂度。为此,我们提出Adaptive-VoCo,增强VoCo-LLaMA以实现自适应压缩。该框架引入轻量级预测器,利用视觉编码器的统计线索(如补丁令牌熵、注意力图方差)量化图像复杂度,并动态选择最优压缩率。此外,设计联合损失函数,融合速率正则化与复杂度对齐,使模型在推理效率与表征能力间取得平衡,尤其在挑战性场景中表现更优。实验表明,本方法在多个多模态任务中持续超越固定率基线,凸显自适应视觉压缩在构建更高效、鲁棒的VLMs中的潜力。

原文摘要 · Abstract (English)

In recent years, large-scale vision-language models (VLMs) have demonstrated remarkable performance on multimodal understanding and reasoning tasks. However, handling high-dimensional visual features often incurs substantial computational and memory costs. VoCo-LLaMA alleviates this issue by compressing visual patch tokens into a few VoCo tokens, reducing computational overhead while preserving strong cross-modal alignment. Nevertheless, such approaches typically adopt a fixed compression rate, limiting their ability to adapt to varying levels of visual complexity. To address this limitation, we propose Adaptive-VoCo, a framework that augments VoCo-LLaMA with a lightweight predictor for adaptive compression. This predictor dynamically selects an optimal compression rate by quantifying an image's visual complexity using statistical cues from the vision encoder, such as patch token entropy and attention map variance. Furthermore, we introduce a joint loss function that integrates rate regularization with complexity alignment. This enables the model to balance inference efficiency with representational capacity, particularly in challenging scenarios. Experimental results show that our method consistently outperforms fixed-rate baselines across multiple multimodal tasks, highlighting the potential of adaptive visual compression for creating more efficient and robust VLMs.

视觉压缩多模态模型自适应推理VLM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。