arXiv:2603.09160cs.CVcs.AI2026-03被引 5

用LLM写评分标准,让模型自动生成更丰富多样的图像描述。

RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioning

  • 用LLM生成评分细则,为每张图定制细粒度奖励信号。
  • 在CapArena上胜过人类标注和GPT-4V,CaptionQA更高效。
  • 小模型3B可训练出比大模型更强的视觉语言模型,适合资源有限者。

密集图像描述对视觉语言预训练和文生图至关重要,但高质量标注成本过高。虽可用强视觉语言模型(VLM)生成合成描述,但监督蒸馏常导致输出多样性不足、泛化能力弱。强化学习(RL)可突破此局限,但其成功多限于有确定性校验器的领域——而开放文本描述缺乏此类条件。我们提出RubiCap,一种新型RL框架,通过LLM编写的评分标准生成细粒度、样本特异的奖励信号。RubiCap先生成多样候选描述,再由LLM评分撰写者提取共识优点并诊断当前策略缺陷,转化为明确评估标准,使LLM裁判能分解整体质量评价,以结构化多维度评估替代粗略标量奖励。在多个基准测试中,RubiCap在CapArena上胜率最高,超越监督蒸馏、先前RL方法、人类专家标注及GPT-4V增强结果;在CaptionQA上表现更优:7B模型媲美Qwen2.5-VL-32B-Instruct,3B模型超越其7B版本。令人惊讶的是,使用轻量级RubiCap-3B作为描述生成器,所训练的VLM性能优于基于专有模型标注训练的模型。

原文摘要 · Abstract (English)

Dense image captioning is critical for cross-modal alignment in vision-language pretraining and text-to-image generation, but scaling expert-quality annotations is prohibitively expensive. While synthetic captioning via strong vision-language models (VLMs) is a practical alternative, supervised distillation often yields limited output diversity and weak generalization. Reinforcement learning (RL) could overcome these limitations, but its successes have so far been concentrated in verifiable domains that rely on deterministic checkers -- a luxury not available in open-ended captioning. We address this bottleneck with RubiCap, a novel RL framework that derives fine-grained, sample-specific reward signals from LLM-written rubrics. RubiCap first assembles a diverse committee of candidate captions, then employs an LLM rubric writer to extract consensus strengths and diagnose deficiencies in the current policy. These insights are converted into explicit evaluation criteria, enabling an LLM judge to decompose holistic quality assessment and replace coarse scalar rewards with structured, multi-faceted evaluations. Across extensive benchmarks, RubiCap achieves the highest win rates on CapArena, outperforming supervised distillation, prior RL methods, human-expert annotations, and GPT-4V-augmented outputs. On CaptionQA, it demonstrates superior word efficiency: our 7B model matches Qwen2.5-VL-32B-Instruct, and our 3B model surpasses its 7B counterpart. Remarkably, using the compact RubiCap-3B as a captioner produces stronger pretrained VLMs than those trained on captions from proprietary models.

图像描述强化学习LLM评分高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。