通过视觉锚点提示缓解多模态大模型的视觉退化问题
Mitigating Visual Degradation in MLLMs via Spatial-Spectral Visual Anchor Learning
- 引入视觉锚点提示，利用外部视觉模型知识稳定内部表示
- 在中间层加入空间与频域对齐损失，提升视觉表征精度
- 有效解决推理时语义漂移，适合改进视觉感知任务的模型
尽管多模态大语言模型（MLLMs）取得进展，其视觉感知仍存在缺陷。在视觉指令微调后，内部表示在推理过程中迅速偏离原始语义状态，导致严重信息退化。现有方法尝试借助外部视觉基础模型（VFMs）对齐内部表示，但直接对齐会增强视觉语义却无法缓解表示偏差。为此，本文提出空间-谱域视觉锚点学习（SSVAL）。核心是视觉锚点提示注入（VAPI），在训练中引入能吸收外部VFMs丰富知识的提示，使其在推理时作为稳定视觉锚点，抑制表示漂移。此外，还在中间层加入辅助的空间和频域表示对齐损失，提供视觉特异性监督。大量实验表明，SSVAL显著优于现有方法。代码已公开于项目主页。
原文摘要 · Abstract (English)
Despite the progress of multimodal large language models (MLLMs), they continue to exhibit deficiencies in visual perception. Following visual instruction tuning, internal MLLM representations rapidly deviate from their original semantic states during inference, causing severe information degradation. While existing methods attempt to leverage external vision foundation models (VFMs) to align internal representations, we find that direct alignment with VFMs enhances visual semantics but fails to mitigate representation deviation. To address this, we propose Spatial-Spectral Visual Anchor Learning (SSVAL). The core of SSVAL is Visual Anchor Prompt Injection (VAPI), which introduces prompts that absorb rich knowledge from external VFMs during training, enabling them to serve as stable visual anchors that mitigate representation deviation during inference. Additionally, we incorporate auxiliary spatial and frequency-domain representation alignment losses to provide complementary vision-specific supervision at intermediate LLM layers. Extensive experiments demonstrate that SSVAL significantly outperforms existing methods. Code are available on our \href{https://msls38.github.io/SSVAL/}{project page}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。