arXiv:2604.21343cs.CV2026-04

通过噪声修复提升大模型视觉对齐能力,增强跨域泛化性。

Latent Denoising Improves Visual Alignment in Large Multimodal Models

论文配图:Latent Denoising Improves Visual Alignment in Large Multimodal Models
图 1 · 摘自论文原文
  • 用显著性感知的掩码与高斯噪声污染视觉特征,让模型学会恢复原始图像块。
  • 在多个基准测试中显著提升视觉理解与组合推理能力,尤其在自然损坏数据上表现更稳。
  • 无需额外推理开销,适合部署于现有多模态大模型,提升鲁棒性。

大型多模态模型(如 LLaVA)通常采用自回归语言建模目标训练,仅对视觉标记提供间接监督,导致内部视觉表征弱且在分布外时行为脆弱。受潜在空间去噪学习高质量视觉分词器的启发,我们证明该原理可有效提升多模态模型的视觉特征对齐与跨模态理解。提出一种潜在去噪框架:使用显著性感知的混合掩码与高斯噪声破坏投影后的视觉标记,让模型从选定中间层的隐藏状态中通过解码器恢复干净的教师图像块特征。为防止表征坍缩,框架还保留教师模型内的图像内相似性结构,并应用图像内对比式块蒸馏。推理时关闭噪声注入与辅助头,不增加计算开销。在广泛的标准多模态基准测试中,本方法持续优于强基线,在组合鲁棒性测试(如 NaturalBench)中取得显著提升。在 ImageNet-C 类非对抗性常见损坏下,模型保持更高准确率,且在中度与重度损坏下退化更小。代码已开源:https://github.com/dhruvashp/latent-denoising-for-lmms。

原文摘要 · Abstract (English)

Large Multimodal Models (LMMs) such as LLaVA are typically trained with an autoregressive language modeling objective, providing only indirect supervision to visual tokens. This often yields weak internal visual representations and brittle behavior under distribution shift. Inspired by recent progress on latent denoising for learning high-quality visual tokenizers, we show that the same principle provides an effective form of visual supervision for improving internal visual feature alignment and multimodal understanding in LMMs. We propose a latent denoising framework that corrupts projected visual tokens using a saliency-aware mixture of masking and Gaussian noising. The LMM is trained to denoise these corrupted tokens by recovering clean teacher patch features from hidden states at a selected intermediate LLM layer using a decoder. To prevent representation collapse, our framework also preserves the teacher's intra-image similarity structure and applies intra-image contrastive patch distillation. During inference, corruption and auxiliary heads are disabled, introducing no additional inference-time overhead. Across a broad suite of standard multimodal benchmarks, our method consistently improves visual understanding and reasoning over strong baselines, and yields clear gains on compositional robustness benchmarks (e.g., NaturalBench). Moreover, under ImageNet-C-style non-adversarial common corruptions applied to benchmark images, our method maintains higher accuracy and exhibits reduced degradation at both moderate and severe corruption levels. Our code is available at https://github.com/dhruvashp/latent-denoising-for-lmms.

多模态去噪视觉对齐鲁棒性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。