arXiv:2606.08063cs.CVcs.AI2026-06中稿 · ICML被引 1

让多模态大模型自修复损坏图像,提升真实场景下的理解能力。

Robust-U1: Can MLLMs Self-Recover Corrupted Visual Content for Robust Understanding?

论文配图:Robust-U1: Can MLLMs Self-Recover Corrupted Visual Content for Robust Understanding?
图 1 · 摘自论文原文
  • 给大模型加入自修复视觉内容的能力,分三步训练:重建、强化学习优化质量、联合推理。
  • 在真实世界图像损坏测试中表现领先,对抗性干扰下仍保持高精度。
  • 适合关注模型鲁棒性、视觉恢复与多模态推理的开发者和研究者。

多模态大语言模型(MLLMs)在视觉理解上表现卓越,但在真实世界视觉退化情况下性能显著下降。现有增强鲁棒性的方法存在局限:黑箱特征对齐缺乏可解释性,白箱文本推理无法恢复像素级细节。本文提出新问题:MLLMs能否自我修复受损视觉内容?为此,我们设计了Robust-U1框架,赋予模型显式的视觉自恢复能力。该方法包含三个阶段:监督微调实现初步重建,基于像素级SSIM和语义级CLIP相似度双重奖励的强化学习提升视觉质量,以及结合原始输入与恢复图像的联合多模态推理。大量实验表明,Robust-U1在真实世界退化基准上达到当前最优表现,并在通用VQA基准的对抗性干扰下保持优异性能。分析证实,高质量视觉恢复直接提升推理准确率,确立自恢复是鲁棒视觉理解的关键机制。代码已开源。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) have demonstrated remarkable success in visual understanding, yet their performance degrades significantly under real-world visual corruptions. While existing robustness enhancement approaches exist, they are limited: black-box feature alignment lacks interpretability, and white-box text-based reasoning cannot restore lost pixel-level details. This work investigates a fundamental research question: Can MLLMs recover corrupted visual content by themselves? To address this, we propose Robust-U1, a novel framework that equips MLLMs with explicit visual self-recovery capability for robust understanding. The approach comprises three core stages: supervised fine-tuning for initial reconstruction, reinforcement learning with dual rewards (pixel-level SSIM and semantic-level CLIP similarity) for aligning high visual quality, and multimodal reasoning that jointly considers both the corrupted input and the recovered image. Extensive experiments demonstrate that Robust-U1 achieves state-of-the-art robustness on the real-world corruption benchmark and maintains superior performance under adversarial corruptions on general VQA benchmarks. Analysis confirms that high-quality visual recovery directly enhances reasoning performance, establishing self-recovery as a critical mechanism for robust visual understanding. The source code is available at https://github.com/jqtangust/Robust-U1.

多模态鲁棒性自修复视觉理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。