arXiv:2506.07971cs.CV2025-06被引 7

让视频大模型像生物一样自我调节,提升理解长视频的准确性和稳定性。

CyberV: Cybernetics for Test-time Scaling in Video Understanding

  • 引入控制论循环机制,让模型在推理时自监控、自纠错并动态分配资源。
  • 在VideoMMMU上使小模型性能提升8.3%,大模型达10.0%,接近人类水平。
  • 无需重训练或新增模块,适合部署在资源受限的实时视频理解场景。

当前多模态大语言模型在理解长视频或复杂视频时面临计算开销大、鲁棒性差、精度不足等问题,主要源于其前向传播的固有特性,且对参数较少的模型影响更显著。为此,我们提出受控制论启发的新框架CyberV,将视频多模态大模型重构为具备自我监控、自我修正和动态资源分配能力的自适应系统。该框架包含一个推理系统、传感器与控制器组成的闭环:传感器监测模型前向过程,收集注意力漂移等中间表征;控制器据此判断何时及如何触发自我修正,并生成反馈指导下一轮推理。此测试时自适应扩展方法无需重训练或添加额外组件。实验表明,CyberV在VideoMMMU上使Qwen2.5-VL-7B提升8.3%,InternVL3-8B提升5.5%,超越竞品GPT-4o;应用于Qwen2.5-VL-72B时实现10.0%提升,表现媲美人类专家。同时在VideoMME和WorldSense等通用基准上也展现一致增益,验证了其在动态视频理解中的有效性与泛化能力。代码已开源。

原文摘要 · Abstract (English)

Current Multimodal Large Language Models (MLLMs) may struggle with understanding long or complex videos due to computational demands at test time, lack of robustness, and limited accuracy, primarily stemming from their feed-forward processing nature. These limitations could be more severe for models with fewer parameters. To address these limitations, we propose a novel framework inspired by cybernetic principles, redesigning video MLLMs as adaptive systems capable of self-monitoring, self-correction, and dynamic resource allocation during inference. Our approach, CyberV, introduces a cybernetic loop consisting of an MLLM Inference System, a Sensor, and a Controller. Specifically, the sensor monitors forward processes of the MLLM and collects intermediate interpretations, such as attention drift, then the controller determines when and how to trigger self-correction and generate feedback to guide the next round. This test-time adaptive scaling framework enhances frozen MLLMs without requiring retraining or additional components. Experiments demonstrate significant improvements: CyberV boosts Qwen2.5-VL-7B by 8.3% and InternVL3-8B by 5.5% on VideoMMMU, surpassing the competitive proprietary model GPT-4o. When applied to Qwen2.5-VL-72B, it yields a 10.0% improvement, achieving performance even comparable to human experts. Furthermore, our method demonstrates consistent gains on general-purpose benchmarks, such as VideoMME and WorldSense, highlighting its effectiveness and generalization capabilities in making MLLMs more robust and accurate for dynamic video understanding. The code is released at https://github.com/marinero4972/CyberV.

视频理解控制论自适应推理大模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。