通过自适应深度聚合提升视觉语言模型推理能力，不牺牲感知性能。
Mitigating the Reasoning Tax in Vision-Language Fine-Tuning with Input-Adaptive Depth Aggregation
- 设计输入自适应深度聚合机制，动态融合多层特征。
- 在低秩微调下提升推理得分9.5分，感知得分提升3.3分。
- 仅增加0.14万参数，适合资源受限场景下的高效微调。
在视觉指令数据上进行监督微调（SFT）常能提升视觉语言模型（VLMs）的感知能力，但会损害其推理性能，形成持续存在的推理代价。我们探究该退化是否与深度特征访问受阻有关，发现即使采用固定的跨深度聚合也能显著恢复推理能力，表明保持跨深度特征访问是当前微调方法中缺失的关键因素。基于此，我们提出输入自适应深度聚合（IADA），一种轻量级机制，使跨深度检索具备输入自适应、模态感知特性，并通过低秩瓶颈高效参数化。在Qwen3-VL-2B上，IADA相较于仅使用LoRA的微调，平均推理得分提升9.5分，感知得分提升3.3分，仅增加0.14M额外参数，且在参数高效的低秩设置下收益最显著。
原文摘要 · Abstract (English)
Supervised fine-tuning (SFT) on visual instruction data often improves perceptual capabilities in vision-language models (VLMs) while degrading reasoning performance, creating a persistent reasoning tax during post-training. We investigate whether this degradation is related to disrupted access to depth-wise representations, and find that even fixed cross-depth aggregation substantially restores reasoning, suggesting that preserved cross-depth access is an important missing factor in VLM fine-tuning. Building on this observation, we propose Input-Adaptive Depth Aggregation (IADA), a lightweight mechanism that makes cross-depth retrieval input-adaptive, modality-aware, and efficiently parameterized through a low-rank bottleneck. On Qwen3-VL-2B, IADA improves the average reasoning score by 9.5 points and the average perception score by $3.3$ points over LoRA-only fine-tuning with only 0.14M additional parameters, with the strongest gains appearing in parameter-efficient low-rank settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。