arXiv:2602.22013cs.CV2026-02中稿 · CVPR

让视觉问答模型在模糊、低光等差画质下仍准确,靠因果分离提升鲁棒性。

RobustVisRAG: Causality-Aware Vision-Based Retrieval-Augmented Generation under Visual Degradations

  • 分两条路径:一条捕捉干扰信息,一条提取干净语义,实现解耦。
  • 在真实退化图像上,检索和生成性能提升超6%,端到端效果提高12.4%。
  • 适合做视觉问答的开发者,尤其关注实际场景中画质下降的问题。

基于视觉的检索增强生成(VisRAG)利用视觉语言模型(VLMs)联合检索相关视觉文档并生成基于多模态证据的可信答案。然而,当视觉输入受模糊、噪声、低光或阴影等失真影响时,现有VisRAG模型性能显著下降,因语义与退化因素在预训练视觉编码器中纠缠,导致检索与生成阶段均出错。为此,本文提出RobustVisRAG,一种基于因果引导的双路径框架,在保持效率和零样本泛化能力的同时提升视觉鲁棒性。该框架通过非因果路径以单向注意力捕获退化信号,因果路径则在退化信号引导下学习纯净语义。结合提出的非因果退化建模与因果语义对齐目标,实现语义与退化的清晰分离,使系统在恶劣视觉条件下仍能稳定检索与生成。为评估真实场景下的鲁棒性,我们构建了Distortion-VisRAG数据集,包含跨七领域的大规模合成与真实退化文档,涵盖12种合成与5种真实退化类型,全面反映实际视觉退化。实验结果表明,RobustVisRAG在真实退化图像上,检索、生成及端到端性能分别提升7.35%、6.35%和12.40%,且在清晰输入上保持相近准确率。

原文摘要 · Abstract (English)

Vision-based Retrieval-Augmented Generation (VisRAG) leverages vision-language models (VLMs) to jointly retrieve relevant visual documents and generate grounded answers based on multimodal evidence. However, existing VisRAG models degrade in performance when visual inputs suffer from distortions such as blur, noise, low light, or shadow, where semantic and degradation factors become entangled within pretrained visual encoders, leading to errors in both retrieval and generation stages. To address this limitation, we introduce RobustVisRAG, a causality-guided dual-path framework that improves VisRAG robustness while preserving efficiency and zero-shot generalization. RobustVisRAG uses a non-causal path to capture degradation signals through unidirectional attention and a causal path to learn purified semantics guided by these signals. Together with the proposed Non-Causal Distortion Modeling and Causal Semantic Alignment objectives, the framework enforces a clear separation between semantics and degradations, enabling stable retrieval and generation under challenging visual conditions. To evaluate robustness under realistic conditions, we introduce the Distortion-VisRAG dataset, a large-scale benchmark containing both synthetic and real-world degraded documents across seven domains, with 12 synthetic and 5 real distortion types that comprehensively reflect practical visual degradations. Experimental results show that RobustVisRAG improves retrieval, generation, and end-to-end performance by 7.35%, 6.35%, and 12.40%, respectively, on real-world degradations, while maintaining comparable accuracy on clean inputs.

视觉问答鲁棒性因果推理多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。