arXiv:2511.08909cs.CV2025-11被引 1

用合成图像抑制错误实体,提升零样本图像描述跨域准确性

Negative Entity Suppression for Zero-Shot Captioning with Synthetic Images

  • 用合成图像统一训练与推理的图文检索,确保一致性
  • 过滤检索结果中的无关实体,降低幻觉率
  • 在注意力层抑制错误实体特征,适合跨域场景研究者

仅用文本训练可缓解零样本图像描述(ZIC)中的数据稀缺问题,避免收集成对图文标注的成本。然而,这类方法在训练域内表现良好,跨域泛化能力差,面对新视觉环境时常产生幻觉内容。基于检索的方法试图缓解此问题,但当检索到的描述包含与输入无关的实体时,反而加剧幻觉。本文提出负向实体概念——即出现在生成描述中但未在输入图像中出现的对象,并引入负向实体抑制(NES)来解决该问题。NES融合三个阶段:(1) 利用合成图像保证训练与推理阶段图像-文本检索的一致性;(2) 过滤检索内容中的负向实体以提高准确性;(3) 使用识别出的负向实体在注意力层面进行抑制,进一步降低幻觉敏感特征的影响。多基准测试表明,NES在保持域内性能的同时,显著提升跨域迁移能力并降低幻觉率,达到新的最先进水平。代码已公开于https://github.com/nidongpinyinme/NESCap。

原文摘要 · Abstract (English)

Text-only training provides an attractive approach to address data scarcity challenges in zero-shot image captioning (ZIC), avoiding the expense of collecting paired image-text annotations. However, although these approaches perform well within training domains, they suffer from poor cross-domain generalization, often producing hallucinated content when encountering novel visual environments. Retrieval-based methods attempt to mitigate this limitation by leveraging external knowledge, but they can paradoxically exacerbate hallucination when retrieved captions contain entities irrelevant to the inputs. We introduce the concept of negative entities--objects that appear in generated caption but are absent from the input--and propose Negative Entity Suppression (NES) to tackle this challenge. NES seamlessly integrates three stages: (1) it employs synthetic images to ensure consistent image-to-text retrieval across both training and inference; (2) it filters negative entities from retrieved content to enhance accuracy; and (3) it applies attention-level suppression using identified negative entities to further minimize the impact of hallucination-prone features. Evaluation across multiple benchmarks demonstrates that NES maintains competitive in-domain performance while improving cross-domain transfer and reducing hallucination rates, achieving new state-of-the-art results in ZIC. Our code is available at https://github.com/nidongpinyinme/NESCap.

零样本描述幻觉抑制合成图像跨域泛化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。