通过多智能体评分与共识重排，实现无需训练的零样本图像描述生成突破。
Adjudicated Captioning: Multi-Agent Alignment Scoring and Consensus-Distilled Beam Arbitration for Strict Zero-Shot Image Captioning
- 引入多阶段视觉对齐评分机制，恢复解码过程中的视觉反馈。
- 在COCO上达到117.6的CIDEr和21.9的SPICE，较基线提升9.6点。
- 完全不重训练生成器，适合追求高精度零样本生成的研究者。
零样本图像描述（ZIC）在训练时无需图像-文本配对监督，仅依赖纯文本语料和冻结的预训练图文评分器。现有检索增强方法仅在检索阶段评分一次，随后仅凭语言模型概率决定解码路径，导致生成过程缺乏持续视觉对齐反馈。自2024年以来，该领域进展停滞。本文提出“仲裁式描述”（Adjudicated Captioning），一种推理时的多智能体框架，在不改变IFCap生成器的前提下，于多个关键节点恢复视觉反馈。首先，在输入端部署更强的冻结检索编码器；其次，在检索与解码之间加入冻结的交叉注意力验证器，将前9个候选重排至前5个；最后，在输出束上引入可学习的重排器，由TriFuse（MLP）与MemAttend（记忆注意力Transformer）构成，二者通过三重冻结评分器间的博尔达共识蒸馏进行自监督训练，不使用任何配对数据或参考描述。在归纳头协议下，基于独立的COCO Karpathy验证集训练重排器并冻结应用于测试，框架在COCO Karpathy上取得117.6的CIDEr和21.9的SPICE，相比IFCap的108.0和20.3提升9.6点，优于最强合成图像增强方法NES的109.9。训练无关的固定融合基线达115.8，其中7.8点来自非学习架构改进，1.8点来自可学习重排器。该方案无需重训练生成器即可迁移到外部数据集：Flickr30k Karpathy提升8.1，NoCaps整体提升5.7。
原文摘要 · Abstract (English)
Zero-shot image captioning (ZIC) describes images without paired image-caption supervision during captioner training, relying on text-only corpora and frozen pretrained image-text scorers. Existing retrieval-augmented methods score image-text alignment once, at retrieval, then commit the captioner's autoregressive beam under language-model probability alone, leaving the decoder without further visual grounding feedback. Progress has stalled, with no method improving on the strict-regime best since 2024. We propose Adjudicated Captioning, an inference-time multi-agent framework that restores grounding feedback at multiple checkpoints over an unchanged IFCap captioner. First, we install a stronger frozen Retrieval Encoder at the input. Second, between retrieval and decoding we insert a frozen Cross-Attention Verifier that re-ranks the top-9 retrievals to top-5. Third, at the output beam we attach a learned Reranker pairing TriFuse, a multilayer perceptron, with MemAttend, a memory-attended transformer, the pipeline's only learned components; both are trained self-supervised by Borda-consensus distillation across the three frozen scorers, using no paired image-caption labels and no reference captions. Under the inductive headline protocol, with rerankers fit on the disjoint COCO Karpathy validation beam and applied frozen to test, the framework reaches CIDEr 117.6 and SPICE 21.9 on COCO Karpathy, up from 108.0 and 20.3 for IFCap, a +9.6 CIDEr gain, and +7.7 above NES, the strongest synthetic-image-augmented method at 109.9, without retraining the captioner. A training-free fixed-fusion baseline reaches 115.8 CIDEr, so +7.8 of the +9.6 gain comes from the non-learned architectural intervention and the remaining +1.8 from the learned rerankers. The same recipe transfers off-COCO without captioner retraining: +8.1 CIDEr on Flickr30k Karpathy and +5.7 on NoCaps overall.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。