解决MLLM OCR中低层特征被梯度干扰的问题，提升文字识别精度。
Detached Skip-Links and $R$-Probe: Decoupling Feature Aggregation from Gradient Propagation for MLLM OCR
- 用非对称跳连接分离特征融合与梯度传播，避免低层信息被覆盖。
- 在700万样本上测试，多个ViT模型和基准下OCR性能均提升。
- 适合需要高精度视觉细节理解的多模态任务，如文档理解。
多模态大语言模型（MLLM）在高层次推理上表现优异，但在光学字符识别（OCR）任务中因细粒度视觉细节丢失或错位而表现不佳。我们发现多层特征融合中存在被忽视的优化问题：跳跃路径会将高层语义目标的梯度直接反传至早期视觉层，覆盖低层信号并导致训练不稳定。为此，我们提出Detached Skip-Links，一种极简改进方案——在前向传播中复用浅层特征，但训练时阻断跳跃分支的梯度。该不对称设计有效降低梯度干扰，提升训练稳定性和收敛性，且无需增加可学习参数。为诊断低层信息是否被LLM保留并可用，我们引入$R$-Probe，使用从LLM前四分之一层初始化的浅层解码器，测量投影后视觉标记的像素级重建能力。在多个ViT骨干网络和多模态基准上，涵盖高达700万训练样本，本方法在以OCR为核心的基准上持续提升，并在通用多模态任务中带来显著增益。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) excel at high-level reasoning yet fail on OCR tasks where fine-grained visual details are compromised or misaligned. We identify an overlooked optimization issue in multi-layer feature fusion. Skip pathways introduce direct back-propagation paths from high-level semantic objectives to early visual layers. This mechanism overwrites low-level signals and destabilizes training. To mitigate this gradient interference, we propose Detached Skip-Links, a minimal modification that reuses shallow features in the forward pass while stopping gradients through the skip branch during joint training. This asymmetric design reduces gradient interference, improving stability and convergence without adding learnable parameters. To diagnose whether fine-grained information is preserved and usable by an LLM, we introduce $R$-Probe, which measures pixel-level reconstructability of projected visual tokens using a shallow decoder initialized from the first quarter of the LLM layers. Across multiple ViT backbones and multimodal benchmarks, and at scales up to 7M training samples, our approach consistently improves OCR-centric benchmarks and delivers clear gains on general multimodal tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。