用CLIP引导测试时训练，让大模型在图像受损时更少幻觉。
ClipTTT: CLIP-Guided Test-Time Training Helps LVLMs See Better
- 用CLIP的图文对齐能力做自监督信号，实时调整模型。
- 在15种常见图像退化下，幻觉率显著下降，描述更忠实。
- 适合部署在真实场景中图像质量不稳定的视觉语言任务。
大型视觉语言模型（LVLM）在测试时遇到图像退化容易产生幻觉。我们发现这些退化相当于额外的分布偏移，大幅加剧了现实应用中的幻觉现象。为此，提出一种无需修改基础模型的在线适应方法——CLIP引导的测试时训练（ClipTTT）。该方法利用预训练CLIP模型的图像-文本对齐能力，作为稳定的目标信号，快速识别可靠自监督信号，在单个测试样本上实现即时适配。在包含15种常见退化的标准幻觉基准上进行的大量实验表明，ClipTTT能有效缓解幻觉问题，提升描述的忠实度。
原文摘要 · Abstract (English)
Large vision-language models (LVLMs) tend to hallucinate, especially when visual inputs are corrupted at test time. We show that such corruptions act as additional distribution shifts, significantly amplifying hallucination rates in real-world applications. To address this, we propose CLIP-guided Test-Time Training (ClipTTT), a method to adapt LVLMs under degraded conditions on the fly with a single test sample. Specifically, we leverage the image-text alignment strength of a pre-trained CLIP model as a stable guidance signal to identify reliable self-supervision targets, enabling rapid adaptation without altering the base LVLMs. Extensive experiments on standard hallucination benchmarks, with 15 common corruptions, demonstrate that ClipTTT effectively mitigates hallucinations and improves descriptive faithfulness under visual corruptions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。