提出细粒度视频问答数据质量评估方法EVQAScore，无需参考答案即可高效评估长视频质量。
EVQAScore: A Fine-grained Metric for Video Question Answering Data Quality Evaluation
- 基于关键词提取与帧采样，实现无参考的视频问答数据质量评估
- 在VATEX-EVAL上达SOTA，Kendall相关性32.8，比前人高4.7
- 仅用12.5%数据量即超越全量数据训练效果，适合高效数据筛选
视频问答（Video QA）是视频理解的核心任务。评估用于训练视频大语言模型（VideoLLMs）的视频问答和视频字幕数据质量是一项关键挑战。尽管已有多种方法用于评估视频字幕质量，但针对视频问答数据缺乏专用评估方法。为此，我们提出EVQAScore，一种无需参考答案的方法，利用关键词提取评估视频字幕与视频问答数据质量。此外，通过引入帧采样与重缩放技术，提升了评估效率与鲁棒性，使该评分可处理极长视频。在VATEX-EVAL基准上，该方法在视频字幕评估中达到当前最优性能（肯德尔相关性32.8，斯皮尔曼相关性42.3），分别较前人方法PAC-S++提升4.7和5.9。进一步地，使用EVQAScore进行数据筛选，仅需原始数据量的12.5%即可取得优于全量数据训练的SOTA结果，显著超越此前最佳方法PAC-S。
原文摘要 · Abstract (English)
Video question-answering (QA) is a core task in video understanding. Evaluating the quality of video QA and video caption data quality for training video large language models (VideoLLMs) is an essential challenge. Although various methods have been proposed for assessing video caption quality, there remains a lack of dedicated evaluation methods for Video QA. To address this gap, we introduce EVQAScore, a reference-free method that leverages keyword extraction to assess both video caption and video QA data quality. Additionally, we incorporate frame sampling and rescaling techniques to enhance the efficiency and robustness of our evaluation, this enables our score to evaluate the quality of extremely long videos. Our approach achieves state-of-the-art (SOTA) performance (32.8 for Kendall correlation and 42.3 for Spearman correlation, 4.7 and 5.9 higher than the previous method PAC-S++) on the VATEX-EVAL benchmark for video caption evaluation. Furthermore, by using EVQAScore for data selection, we achieved SOTA results with only 12.5\% of the original data volume, outperforming the previous SOTA method PAC-S and 100\% of data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。