不依赖外部检索库，用触发增强与多模态净化实现零样本图像描述生成。
TPCap: Unlocking Zero-Shot Image Captioning with Trigger-Augmented and Multi-Modal Purification Modules
- 通过触发投影激活大模型上下文推理能力，提升图文对齐效果。
- 仅用0.82M参数，在单张4090显卡上训练，性能媲美顶尖模型。
- 适合追求轻量级、高效率零样本图像描述生成的研究者与应用者。
近年来，大语言模型（LLMs）显著提升了图像描述的流畅性与逻辑连贯性。检索增强生成（RAG）被广泛用于向LLMs引入外部知识，但现有方法依赖独立的检索库，带来计算开销，并限制了LLMs固有的零样本能力。为解决这些问题，我们提出TPCap：一种无需外部检索库的触发增强与多模态净化框架，用于零样本图像描述生成。TPCap包含两个核心模块：触发增强（TA）和多模态净化（MP）。TA模块使用冻结与可学习的投影触发器，激活LLMs的上下文推理，增强视觉-文本对齐，缓解数据偏差。MP模块通过过滤噪声与提升特征质量，进一步优化生成实体信息，确保描述更精确且事实一致。我们在COCO、NoCaps、Flickr30k和WHOOPS数据集上评估了TPCap。仅需0.82M可训练参数，并在单张NVIDIA RTX 4090 GPU上训练，其性能达到当前最优水平。
原文摘要 · Abstract (English)
Recent advancements in large language models (LLMs) have significantly enhanced the fluency and logical coherence of image captioning. Retrieval-Augmented Generation (RAG) is widely adopted to incorporate external knowledge into LLMs; however, existing RAG-based methods rely on separate retrieval banks, introducing computational overhead and limiting the utilization of LLMs' inherent zero-shot capabilities. To address these limitations, we propose TPCap, a novel trigger-augmented and multi-modal purification framework for zero-shot image captioning without external retrieval libraries. TPCap consists of two key components: trigger-augmented (TA) generation and multi-modal purification (MP). The TA module employs a trigger projector with frozen and learnable projections to activate LLMs' contextual reasoning, enhance visual-textual alignment, and mitigate data bias. The MP module further refines the generated entity-related information by filtering noise and enhancing feature quality, ensuring more precise and factually consistent captions. We evaluate TPCap on COCO, NoCaps, Flickr30k, and WHOOPS datasets. With only 0.82M trainable parameters and training on a single NVIDIA RTX 4090 GPU, TPCap achieves competitive performance comparable to state-of-the-art models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。