为视觉语言模型设计可迁移的精简视觉路径，提升推理效率
Searching for Task-Specific Vision Paths: Evolutionary Block Pruning Across Vision-Language Models
- 用进化算法搜索共享或能力特定的视觉块组合路径
- 在相同计算预算下，搜索路径比随机/连续删除更优，最高提升4.91点
- 跨模型路径可迁移，但能力标签不保证通用性，OCR任务例外
视觉语言模型通常对每个问题都运行完整的视觉编码器，即使不同任务（如OCR、计数、物体识别）所需计算量不同。本文研究在不微调的前提下，能否通过固定预算的视觉块组合跳过部分计算。提出一种共享K块路径和基于能力标签的K块策略，并采用源平衡的进化搜索方法，与独立排序、连续移除和随机路径在相同预算下对比。实验使用Qwen2.5-VL-3B-Instruct、SmolVLM2-2.2B-Instruct及876例图像不重叠的数据集。搜索路径在不同架构间可迁移：在SmolVLM2上，共享四块路径比独立构建高4.91个百分点；在Qwen上，六块能力专用路径优于共享路径2.17点，其中OCR任务提升7.10点；但在封闭数据集IIIT5K上，其专用于OCR的路径反而落后共享路径13.6点。组合搜索显著提升路径质量，但能力标签无法定义通用视觉路径。
原文摘要 · Abstract (English)
Vision-language models normally execute the same complete vision encoder for every question, even when OCR, counting, object, attribute, and spatial queries may not require identical computation. We study whether fixed-budget combinations of vision blocks can be skipped without fine-tuning. A shared K-block route skips one searched set of exactly K blocks for every question, while a capability-specific K-block policy selects one same-size route using a known capability label. We introduce a source-balanced evolutionary search and compare it with independent ranking, contiguous removal, and random routes at matched budgets. Experiments use Qwen2.5-VL-3B-Instruct, SmolVLM2-2.2B-Instruct, and an 876-example image-disjoint selection split. Search transfers across architectures: on SmolVLM2, the searched shared four-block route beats independent construction by 4.91 percentage points. Capability specialization is less stable. On Qwen, the six-block capability policy beats the shared route by 2.17 points, driven by a 7.10-point OCR gain. On sealed IIIT5K, however, the SmolVLM2 OCR-specific route trails its shared route by 13.6 points. Combinatorial search reliably improves route construction, but capability labels do not define universally transferable vision pathways.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。