用代码复杂度量化视频问答难题，比人判断更准。
Understanding Complexity in VideoQA via Visual Program Generation
- 用生成代码的复杂度衡量视频问答问题难度。
- 新基准比NExT-QA难1.9倍，模型表现显著下降。
- 可自动扩展，适合评估新模型的挑战性能力。
我们提出一种数据驱动的方法，用于分析视频问答（VideoQA）中问题的复杂度。以往的评测设计依赖人工经验构造难题，但我们实验表明人类难以预测机器学习模型实际遇到的困难问题。本方法利用视觉问答中的代码生成技术，以生成代码的复杂度作为问题难度的代理指标，发现该指标与模型性能的相关性显著优于人工判断。为实现这一洞察，我们提出一种从代码估计问题复杂度的算法，能识别出与特定模型最难问题相关的细粒度操作单元，便于未来扩展至新方法。最后，为验证方法实用性，我们将其用于自动生成复杂问题，构建了一个新基准，其难度是流行基准NExT-QA的1.9倍。
原文摘要 · Abstract (English)
We propose a data-driven approach to analyzing query complexity in Video Question Answering (VideoQA). Previous efforts in benchmark design have relied on human expertise to design challenging questions, yet we experimentally show that humans struggle to predict which questions are difficult for machine learning models. Our automatic approach leverages recent advances in code generation for visual question answering, using the complexity of generated code as a proxy for question difficulty. We demonstrate that this measure correlates significantly better with model performance than human estimates. To operationalize this insight, we propose an algorithm for estimating question complexity from code. It identifies fine-grained primitives that correlate with the hardest questions for any given set of models, making it easy to scale to new approaches in the future. Finally, to further illustrate the utility of our method, we extend it to automatically generate complex questions, constructing a new benchmark that is 1.9 times harder than the popular NExT-QA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。