动态调整视觉推理与上下文,让多模态模型更省力更准
AVIS: Adaptive Test-Time Scaling for Vision-Language Models

- 按查询自适应调节视觉上下文和推理深度
- 减少冗余视觉标记,推理次数随难度动态调整
- 无需训练,兼容共享预填充,适合部署
现代多模态模型通过思维链提示和测试时扩展提升性能,但高推理成本源于大视觉上下文和长解码序列。本文从两个耦合维度审视:视觉上下文扩展(VCS)控制输入语言模型的视觉证据量,视觉推理扩展(VRS)控制推理时的搜索强度。现有方法常单一优化其一,未充分探索联合计算分配。提出自适应视觉推理扩展(AVIS),轻量级策略按查询动态调节两者。通过无训练的键多样性视觉(KDV)剪枝实现VCS,$O(N)$复杂度移除冗余视觉标记;通过自适应自一致性实现VRS,利用学习到的难度预测器决定推理轮次。AVIS支持共享预填充推理,所有轮次复用一次预填充与键值缓存,部署友好。在多种图像与视频推理基准上,相较仅调VCS或仅调VRS基线,显著改善精度-计算权衡,且在强化学习微调后的多模态模型上仍保持低计算与低延迟。
原文摘要 · Abstract (English)
Modern Vision-Language Models (VLMs) benefit from chain-of-thought prompting and test-time scaling, but these gains often come with prohibitive inference cost due to large visual contexts and long decoding chains. We view this cost through two coupled axes: Visual Context Scaling (VCS), which controls how much visual evidence is passed to the language model, and Visual Reasoning Scaling (VRS), which controls how much inference-time reasoning search is performed. Existing methods typically optimize one axis at a time, leaving the joint allocation of compute across these axes underexplored. We introduce Adaptive Visual Inference Scaling (AVIS), a lightweight policy that adapts both VCS and VRS per query. AVIS realizes VCS through Key Diversity Visual (KDV) pruning, a training-free $O(N)$ key-based rule for removing redundant visual tokens before prefilling, and realizes VRS through adaptive self-consistency, using a learned difficulty predictor to select the number of reasoning rollouts. AVIS is deployment-friendly and compatible with shared-prefill inference, where all rollouts reuse a single prefilling pass and KV cache. Across diverse image and video reasoning benchmarks, AVIS improves the accuracy--compute trade-off relative to VCS-only and VRS-only baselines, and remains effective on top of RL post-trained VLMs while keeping compute and latency low.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。