用自举方法训练多模态模型评估视频理解输出，效果优于更大模型。
VideoJudge: Bootstrapping Enables Scalable Supervision of MLLM-as-a-Judge for Video Understanding
- 通过生成器与判别器迭代训练，自举构建视频评估模型。
- 7B版在三个基准上超越32B/72B大模型，证明小模型有效。
- 视频输入对评估至关重要，纯语言模型表现更差。
精确评估视频理解模型仍具挑战：常用指标如BLEU、ROUGE和BERTScore难以捕捉人类判断的细微差别，而人工评估成本高昂。近期研究尝试使用大语言模型（LLM）或多模态大语言模型（MLLM）作为评估者，但其在视频理解中的应用仍不充分。本文提出VideoJudge，一个3B和7B规模的专用MLLM评估模型，用于评判视频理解模型的输出（即基于视频的文本响应）。训练采用生成器与评估器协同机制：生成器根据目标评分生成响应，不匹配评估结果的样本被丢弃。在四个元评估基准中的三个，VideoJudge-7B的表现优于更大的MLLM基线（如Qwen2.5-VL，32B和72B）。值得注意的是，仅使用文本的LLM评估器（Qwen3）表现反而劣于带视觉输入的MLLM（Qwen2.5-VL），且长链思维推理并未提升性能，表明视频输入对视频理解任务评估至关重要。
原文摘要 · Abstract (English)
Precisely evaluating video understanding models remains challenging: commonly used metrics such as BLEU, ROUGE, and BERTScore fail to capture the fineness of human judgment, while obtaining such judgments through manual evaluation is costly. Recent work has explored using large language models (LLMs) or multimodal LLMs (MLLMs) as evaluators, but their extension to video understanding remains relatively unexplored. In this work, we introduce VideoJudge, a 3B and 7B-sized MLLM judge specialized to evaluate outputs from video understanding models (\textit{i.e.}, text responses conditioned on videos). To train VideoJudge, our recipe builds on the interplay between a generator and an evaluator: the generator is prompted to produce responses conditioned on a target rating, and responses not matching the evaluator's rating are discarded. Across three out of four meta-evaluation benchmarks, VideoJudge-7B outperforms larger MLLM judge baselines such as Qwen2.5-VL (32B and 72B). Notably, we find that LLM judges (Qwen3) models perform worse than MLLM judges (Qwen2.5-VL) and long chain-of-thought reasoning does not improve performance, indicating that providing video inputs is crucial for evaluation of video understanding tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。