用递归搜索自动优化视频理解的上下文构造程序。
VideoHarness-RSI: Recursive Harness Self-Improvement for Long-Video Understanding with Frozen Vision-Language Models

- 通过递归搜索可执行程序结构,自动改进视频上下文构建方式。
- 相比手工设计基线,提升显著,且在多个长视频数据集上表现更好。
- 方法可迁移,无需重搜即适用于新任务,适合研究模型外优化。
长视频理解的关键在于如何从超长视频中构建有限的模型上下文。现有方法通过压缩、检索、记忆和代理式证据获取来改进这一过程,但这些机制通常作为手动设计的推理系统一部分,或与其他组件联合优化,难以分离出核心问题:仅优化可执行上下文构建程序能带来多大提升?本文提出 VIDEOHARNESS-RSI,一个围绕冻结视觉-语言模型(VLM)的可控基线,通过递归自改善(Recursive Harness Self-Improvement)搜索可执行上下文构造器。外层生成器利用历史程序、评估结果和执行轨迹,生成候选“钩子”(harness),经端到端执行与评估后,成功变体被保留用于进一步搜索。这将长视频理解转化为可控的自动化钩子设计问题:可搜索对象是可执行程序结构,而回答模型和接口保持不变。从均匀采样开始,递归搜索持续发现改进空间,并超越多个弱基线;从更强的手工基线出发,同样实现进一步提升。所选钩子还能在不额外搜索的情况下迁移至其他长视频基准。这些结果确立了可执行上下文构造作为独立优化层,并为冻结VLM上的钩子发现与迁移提供了可复现的基准。
原文摘要 · Abstract (English)
Long-video understanding depends critically on how a limited model context is constructed from a much longer video. Existing approaches improve this process through compression, retrieval, memory, and agentic evidence acquisition, but these mechanisms are typically introduced as part of a manually designed inference system or optimized together with other components. This makes it difficult to isolate a simpler question: how much can be gained by improving the executable context-construction program alone? We study this question through VIDEOHARNESS-RSI, a controlled baseline for recursively searching executable context constructors around a frozen vision-language model (VLM). An outer-loop proposer uses prior programs, evaluation outcomes, and execution traces to generate candidate harnesses, which are executed and evaluated end to end before successful variants are retained for further search. This makes long-video understanding a controlled instance of automated harness design: the searchable object is executable program structure, while the answering model and interface remain fixed. Starting from uniform sampling, recursive harness search consistently finds room for improvement and surpasses several weaker hand-crafted baselines. Starting instead from a stronger hand-crafted baseline, the same RSI process yields a further improvement. The selected harness also transfers to additional long-video benchmarks without further search. Together, these results establish executable context construction as a distinct optimization layer and provide a reproducible baseline for studying harness discovery and transfer around frozen VLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。