用分步规划提升图像描述的细节与连贯性,避免大模型一眼看错。
Top-Down Semantic Refinement for Image Captioning
- 将生成过程建模为分层规划,逐步细化语义。
- 减少大模型调用次数达90%以上,仍保持高质量输出。
- 适合需要精准描述、复杂场景的任务,如细粒度图像理解。
大型视觉语言模型(VLM)在图像描述任务中面临内在矛盾:其强大的单步生成能力常导致目光短浅的决策,难以兼顾全局连贯性与细节丰富性,尤其在需多步推理的复杂场景中更为明显。为此,本文将图像描述重新定义为面向目标的分层精炼规划问题,提出名为自顶向下语义精炼(TDSR)的新框架,将生成过程建模为马尔可夫决策过程(MDP)。针对VLM状态空间庞大带来的计算瓶颈,核心贡献在于设计一种高效蒙特卡洛树搜索(MCTS)算法:通过视觉引导的并行扩展和轻量级价值网络,使对昂贵VLM的调用频率降低一个数量级,且不牺牲规划质量;同时引入自适应提前终止机制,动态匹配计算开销与图像复杂度。在DetailCaps、COMPOSITIONCAP和POPE等多个基准上进行的大量实验表明,TDSR作为即插即用模块,可显著提升现有VLM(如LLaVA-1.5、Qwen2.5-VL)的表现,在细粒度描述、组合泛化和幻觉抑制方面达到或接近当前最优水平。
原文摘要 · Abstract (English)
Large Vision-Language Models (VLMs) face an inherent contradiction in image captioning: their powerful single-step generation capabilities often lead to a myopic decision-making process. This makes it difficult to maintain global narrative coherence while capturing rich details, a limitation that is particularly pronounced in tasks that require multi-step and complex scene description. To overcome this fundamental challenge, we redefine image captioning as a goal-oriented hierarchical refinement planning problem, and further propose a novel framework, named Top-Down Semantic Refinement (TDSR), which models the generation process as a Markov Decision Process (MDP). However, planning within the vast state space of a VLM presents a significant computational hurdle. Our core contribution, therefore, is the design of a highly efficient Monte Carlo Tree Search (MCTS) algorithm tailored for VLMs. By incorporating a visual-guided parallel expansion and a lightweight value network, our TDSR reduces the call frequency to the expensive VLM by an order of magnitude without sacrificing planning quality. Furthermore, an adaptive early stopping mechanism dynamically matches computational overhead to the image's complexity. Extensive experiments on multiple benchmarks, including DetailCaps, COMPOSITIONCAP, and POPE, demonstrate that our TDSR, as a plug-and-play module, can significantly enhance the performance of existing VLMs (e.g., LLaVA-1.5, Qwen2.5-VL) by achieving state-of-the-art or highly competitive results in fine-grained description, compositional generalization, and hallucination suppression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。