提出PARCEL架构,让视觉语言模型在压缩后仍能保持高精度和强空间定位能力。
PARCEL: Pool-Anchored Resampling with Conditioned Elastic Queries for Efficient Vision-Language Understanding
- 用池化锚点+条件化弹性查询,动态分配特征提取任务
- 在27个基准上均优于现有压缩方法,性能效率双提升
- 适合需要灵活部署的高效视觉语言模型应用
大型视觉语言模型将视觉输入映射为密集标记序列,导致推理时存在二次计算瓶颈。弹性视觉标记压缩通过训练单一模型实现多种视觉标记预算下的运行,但现有方法在激进压缩下表现不佳。仅空间压缩(如嵌套池化)表现为不完善的低通滤波器,引发频谱混叠,模糊细粒度细节;仅查询压缩(如嵌套查询重采样)则以非局部摘要替代显式网格对齐标记,严重损害空间定位能力。为解决这一表征冲突,我们提出PARCEL(Pool-Anchored Resampling with Conditioned Elastic Queries for Efficient Vision-Language Understanding),一种视觉标记化架构,通过动态划分特征提取职责:建立池化锚点作为低频布局锚,通过池化条件查询重采样使弹性查询标记依赖于这些锚点,促使查询标记关注互补视觉特征而非冗余空间映射。在27个基准上的广泛评估表明,PARCEL显著提升性能-效率帕累托前沿,在不同视觉标记预算下持续优于现有马特罗什卡基线,同时保持“训练一次,随处部署”的范式。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) map visual inputs into dense token sequences, imposing a quadratic computational bottleneck for inference. Elastic visual-token compression addresses this by training a single model that can run at multiple visual-token budgets. However, existing approaches struggle under aggressive compression. Spatial-only compression, as in nested pooling, behaves as an imperfect low-pass filter and induces spectral aliasing that obscures fine-grained detail. Query-only compression, as in nested query resampling, replaces explicit grid-aligned tokens with non-local summaries and substantially degrades spatial grounding. To resolve this representational conflict, we introduce PARCEL (Pool-Anchored Resampling with Conditioned Elastic Queries for Efficient Vision-Language Understanding), a visual tokenization architecture that dynamically partitions the labor of feature extraction. PARCEL establishes spatial pool tokens as low-frequency layout anchors and conditions elastic query tokens on these anchors through Pool-Conditioned Query Resampling. This encourages query tokens to focus on complementary visual features rather than redundant spatial mapping. Extensive evaluations across 27 benchmarks show that PARCEL improves the performance-efficiency Pareto frontier, consistently outperforming existing matryoshka baselines across visual-token budgets while preserving the "train once, deploy anywhere" paradigm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。