让AI judge更懂图片,通过文字化图像提升评估效果
Bi-Level Prompt Optimization for Multimodal LLM-as-a-Judge
- 将图像转为文本描述,缓解模型上下文限制
- 双层优化同时改进评判指令和图像转写提示
- 在4个数据集上验证,适合多模态内容评估场景
大语言模型已成为评估AI生成内容的主流自动评判工具。尽管表现优异,但其评估结果与人类判断的对齐仍具挑战性。监督微调虽可提升对齐度,但成本高且难以适应新任务。近年来,自动提示优化(APO)提供了一种高效替代方案,可自动优化引导LLM评判的指令。然而,现有APO方法主要针对纯文本评估,在多模态场景中研究不足。本文聚焦多模态LLM作为评判者,尤其针对图像生成内容的评估。我们发现关键瓶颈在于:多模态模型受上下文窗口限制,只能处理有限视觉样本,阻碍了提示的迭代优化。为此,提出BLPO——一种双层提示优化框架,将图像转化为保留评估相关视觉线索的文本表示。该双层优化联合精炼评判提示与图像到文本(I2T)转换提示,在有限上下文预算下保持评估保真度。在四个数据集和三个LLM评判者上的实验表明,该方法有效。
原文摘要 · Abstract (English)
Large language models (LLMs) have become widely adopted as automated judges for evaluating AI-generated content. Despite their success, aligning LLM-based evaluations with human judgments remains challenging. While supervised fine-tuning on human-labeled data can improve alignment, it is costly and inflexible, requiring new training for each task or dataset. Recent progress in auto prompt optimization (APO) offers a more efficient alternative by automatically improving the instructions that guide LLM judges. However, existing APO methods primarily target text-only evaluations and remain underexplored in multimodal settings. In this work, we study auto prompt optimization for multimodal LLM-as-a-judge, particularly for evaluating AI-generated images. We identify a key bottleneck: multimodal models can only process a limited number of visual examples due to context window constraints, which hinders effective trial-and-error prompt refinement. To overcome this, we propose BLPO, a bi-level prompt optimization framework that converts images into textual representations while preserving evaluation-relevant visual cues. Our bi-level optimization approach jointly refines the judge prompt and the I2T prompt to maintain fidelity under limited context budgets. Experiments on four datasets and three LLM judges demonstrate the effectiveness of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。