新评估框架能测出视频是否真正讲懂了论文核心思想
A Good Talk Does not Look Like a Summary, It Teaches You! Measuring Takeaways from Paper-to-Video Talks
- 用新框架检测视频是否讲清了关键概念和逻辑
- 现有生成视频虽覆盖内容但常忽略前置知识解释
- 适合关注教学效果而非形式完整性的研究者
从科学论文自动生成视频正广泛用于教育与科研传播。然而，现有评估指标主要关注视觉质量或论文要点是否出现，未能衡量视频是否真正帮助观众理解核心思想。本文提出EffectivePresentationScorer框架，用于评估科学演讲视频的教学质量，检查其是否清晰解释主观点、引入必要背景概念，并将技术细节与论文核心贡献相连接。应用于现有论文转视频系统后发现，生成视频虽提及正确主题并遵循论文结构，却常未解释先决概念或说明方法为何有效。这类问题在仅关注内容存在性的旧评估中被忽视。
原文摘要 · Abstract (English)
Automatically generated videos from scientific papers are increasingly used for education and research dissemination. However, existing evaluation metrics mainly measure visual quality or whether key points from the paper appear in the video without assessing whether the video actually helps viewers understand the ideas. We introduce EffectivePresentationScorer, a framework for evaluating the instructional quality of scientific presentation videos. It checks whether a video explains the main ideas clearly, introduces needed background concepts, and connects technical details to the main contribution of the paper. When we apply EffectivePresentationScorer to the existing paper-to-video generation systems, we find that generated videos mention the correct topics and follow the structure of the paper but fail to explain prerequisite concepts or clarify why the method works. These failures are often ignored by existing video evaluation metrics, which focus on content presence rather than explanatory quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。