让视觉语言模型学会自我纠错，提升生成准确性
RIV: Recursive Introspection Mask Diffusion Vision Language Model
- 引入自省训练与递归推理机制，识别并修正生成错误
- 在多个基准测试中超越现有模型，实现顶尖性能
- 适合需要高可靠性的多模态生成任务使用
基于掩码扩散的视觉语言模型（MDVLMs）在多模态理解任务中取得了显著进展。然而，这些模型无法纠正生成的词元错误，缺乏自我修正能力。本文提出递归自省掩码扩散视觉语言模型（RIV），通过两种新机制赋予模型自我修正能力。第一是自省训练：引入自省模型识别生成序列中的错误，不仅能检测语法和拼写错误，更重要的是能发现逻辑错误。第二是递归推理：从标准解码开始，由学习到的自省模型识别输出序列中的错误并重新掩码，这一交替过程（解码→自省→重掩码）反复进行，直至获得可靠结果。在多个基准上的实验表明，所提出的RIV达到当前最优性能，显著优于大多数现有MDVLMs。
原文摘要 · Abstract (English)
Mask Diffusion-based Vision Language Models (MDVLMs) have achieved remarkable progress in multimodal understanding tasks. However, these models are unable to correct errors in generated tokens, meaning they lack self-correction capability. In this paper, we propose Recursive Introspection Mask Diffusion Vision Language Model (RIV), which equips the model with self-correction ability through two novel mechanisms. The first is Introspection Training, where an Introspection Model is introduced to identify errors within generated sequences. Introspection Training enables the model to detect not only grammatical and spelling mistakes, but more importantly, logical errors. The second is Recursive Inference. Beginning with the standard unmasking step, the learned Introspection Model helps to identify errors in the output sequence and remask them. This alternating ($\text{unmask}\rightarrow\text{introspection}\rightarrow\text{remask}$) process is repeated recursively until reliable results are obtained. Experimental results on multiple benchmarks demonstrate that the proposed RIV achieves state-of-the-art performance, outperforming most existing MDVLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。