arXiv:2510.13315cs.CVcs.AI2025-10中稿 · ICLR被引 1

让视觉生成更准确:根据问题动态调整图像增强和解码策略

Self-Aug: Query and Entropy Adaptive Decoding for Large Vision-Language Models

  • 根据文本查询自适应生成图像增强,提升语义一致性
  • 根据输出稀疏性动态调节候选词数量,减少幻觉
  • 无需训练,适用于多种大模型,适合需要高准确性的场景

大型视觉语言模型(LVLMs)展现出强大的多模态能力,但继承了底层语言模型的幻觉倾向。尽管已有视觉对比解码方法缓解此问题,但现有方法常使用通用视觉增强,忽略文本查询提供的具体上下文,限制了效果。本文提出一种无需训练的解码策略,包含两项关键贡献:一是自增强提示策略,利用模型内在知识动态对齐查询与视觉增强的语义;二是自适应阈值算法,基于输出稀疏性自适应调整下一个词候选集大小,充分利用对数分布信息。在四个LVLM和七个基准上的大量实验表明,所提解码方法显著提升了事实一致性,优于当前最优解码方法。该工作强调了查询相关增强与熵感知解码对提升LVLM有效生成的重要性。

原文摘要 · Abstract (English)

Large Vision-Language Models (LVLMs) have demonstrated remarkable multimodal capabilities, but they inherit the tendency to hallucinate from their underlying language models. While visual contrastive decoding has been proposed to mitigate this issue, existing methods often apply generic visual augmentations that disregard the specific context provided by the text query, limiting their effectiveness. This study introduces a novel training-free decoding strategy that addresses these limitations, featuring two key contributions. First, a self-augmentation prompting strategy that leverages the intrinsic knowledge of the model to dynamically align semantics between the query and the visual augmentation. Second, an adaptive thresholding algorithm that adaptively adjusts next token candidate size based on the output sparsity, utilizing full information from the logit distribution. Extensive experiments across four LVLMs and seven benchmarks demonstrate that the proposed decoding significantly enhances factual consistency compared to state-of-the-art decoding methods. This work highlights the importance of integrating query-dependent augmentation and entropy-aware decoding for improving effective generation of LVLMs.

视觉语言模型解码优化幻觉抑制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。