通过几何曲率识别关键视频帧,实现高效流式视频理解
CurveStream: Boosting Streaming Video Understanding in MLLMs via Curvature-Aware Hierarchical Visual Memory Management

- 基于特征轨迹曲率动态筛选关键帧,避免冗余存储
- 在多个数据集上提升超10%,最高达13.58%性能增益
- 无需训练,适合实时视频分析与资源受限场景
多模态大语言模型在离线视频理解中表现优异,但面对流式视频时,视觉标记线性膨胀导致内存溢出或灾难性遗忘。现有方法依赖均匀采样、低层物理指标或被动缓存淘汰,缺乏语义感知,易破坏上下文连贯性。为此,我们提出CurveStream——一种无需训练的曲率感知分层视觉记忆管理框架。其核心思想是:连续特征轨迹的高曲率区域与关键全局语义转换高度对齐。基于此,CurveStream通过曲率得分实时评估语义强度,并采用在线K-Sigma动态阈值,将帧自适应分配至清晰与模糊记忆状态,严格控制标记预算。跨多种时间尺度的评估表明,该轻量级框架在StreamingBench上取得10.69%的绝对提升,在OVOBench上达13.58%,显著超越基线,建立流式视频感知新基准。
原文摘要 · Abstract (English)
Multimodal Large Language Models have achieved significant success in offline video understanding, yet their application to streaming videos is severely limited by the linear explosion of visual tokens, which often leads to Out-of-Memory (OOM) errors or catastrophic forgetting. Existing visual retention and memory management methods typically rely on uniform sampling, low-level physical metrics, or passive cache eviction. However, these strategies often lack intrinsic semantic awareness, potentially disrupting contextual coherence and blurring transient yet critical semantic transitions. To address these limitations, we propose CurveStream, a training-free, curvature-aware hierarchical visual memory management framework. Our approach is motivated by the key observation that high-curvature regions along continuous feature trajectories closely align with critical global semantic transitions. Based on this geometric insight, CurveStream evaluates real-time semantic intensity via a Curvature Score and integrates an online K-Sigma dynamic threshold to adaptively route frames into clear and fuzzy memory states under a strict token budget. Evaluations across diverse temporal scales confirm that this lightweight framework, CurveStream, consistently yields absolute performance gains of over 10% (e.g., 10.69% on StreamingBench and 13.58% on OVOBench) over respective baselines, establishing new state-of-the-art results for streaming video perception.The code will be released at https://github.com/streamingvideos/CurveStream.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。