arXiv:2606.09131cs.AIcs.CL2026-06

视觉令牌在中间层就饱和,晚融合可大幅减少计算量。

Late-Layer Fusion is Enough: Dual-Path Vision Token Routing for Multimodal Large Language Models under Visual Saturation

  • 视觉令牌在第4层后不再需要深度处理,用侧路路由到浅层
  • 仅用3%可训练参数,性能与原模型相当,节省大量视觉计算
  • 适合追求高效推理的多模态大模型部署场景

多模态大语言模型通常采用对称的深层Transformer架构,对图像和文本令牌施加相同的计算。然而,图像与文本令牌在信息密度、冗余度和推理深度上存在显著差异。通过对LLaVA-1.5的逐层分析发现,视觉令牌在中层即趋于饱和:文本到图像注意力从第0层的0.68降至第4层的0.07,并在第18层后稳定在0.04左右,而文本令牌仍持续受益于深层语义处理。这表明架构对称性与模态演化深度不同步,导致视觉计算冗余及感知表征漂移。为此,我们提出双路径视觉令牌路由(DPVR),其核心实例DPVR-LF在饱和点将视觉令牌送入单层可训练侧支路,跳过深层图像位置执行十三层纯文本前向传播,并仅在最后层重新融合视觉与文本流。该方法仅需约3%可训练参数,在标准基准上保持竞争力的同时,显著降低深层Transformer中的视觉计算量。结果挑战了视觉令牌必须经过所有深层语言模型层的传统假设,表明单个晚融合层即可维持强感知能力。

原文摘要 · Abstract (English)

Multimodal large language models (MLLMs) commonly inherit the deep, symmetric Transformer backbone designed for unimodal text modeling, and apply the same computation uniformly to image and language tokens. This design overlooks a key modality asymmetry: image and text tokens differ substantially in information density, redundancy, and required reasoning depth. Through a layer-wise analysis of LLaVA-1.5, we observe that vision tokens tend to saturate in the middle layers. Specifically, text-to-image attention decreases from 0.68 at layer 0 to 0.07 by layer 4, and stabilizes near 0.04 after layer 18, whereas text tokens continue to benefit from deep semantic processing. These findings suggest a mismatch between architectural symmetry and depth-asynchronous modality evolution, resulting in redundant visual computation and possible drift in perceptual representations during deep task-specific adaptation. Motivated by this, we propose Dual-Path Vision Token Routing (DPVR), a modality-asymmetric routing framework for efficient MLLMs. Its core instantiation, DPVR-LF (Late-Layer Fusion), routes vision tokens at the saturation point into a one-layer trainable side branch, runs a thirteen-layer text-only forward that skips image positions in the deep stack, and re-fuses the visual and textual streams only at the final layer. With approximately 3% trainable parameters, DPVR-LF preserves competitive multimodal performance on standard benchmarks while reducing visual computation in the deep Transformer stack. The results challenge the conventional assumption that vision tokens must traverse all deep language-model layers, and indicate that a single late fusion layer can be sufficient for maintaining strong perceptual competence in LLaVA-style MLLMs.

多模态视觉路由高效推理晚融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。