针对第一人称视频推理,提出无需训练的高效剪枝方法
EgoPrune: Efficient Token Pruning for Egomotion Video Reasoning in Embodied Agent
- 基于视角变换与冗余过滤,动态剔除重复视觉特征
- 在两个基准上实现更高精度,同时降低70%以上计算开销
- 适合部署在边缘设备上的智能体实时视频处理
第一人称视频是智能体的主要视觉输入,其连续变化的视角对推理效率提出挑战。尽管视觉语言模型具备强大多模态推理能力,但长而冗余的视频输入仍导致高昂计算成本。现有剪枝方法多针对第三人称视频,无法利用第一人称场景中的时空连续性与运动约束。为此,我们提出EgoPrune,一种无需训练的剪枝方法,包含三个模块:从EmbodiedR改进的关键帧选择器,用于时间高效采样;视角感知冗余过滤(PARF),通过视角变换对齐视觉令牌并去除冗余;基于最大边际相关性(MMR)的令牌选择器,兼顾视觉-文本相关性与帧内多样性。在两个第一人称视频基准测试中,EgoPrune在多种剪枝比例下均优于现有无训练方法,显著降低浮点运算量(FLOPs)、内存占用和延迟。此外,我们在搭载Jetson Orin NX 16GB边缘设备的智能体上部署EgoPrune,验证了其实际应用效率,适用于设备端第一人称视频推理。
原文摘要 · Abstract (English)
Egomotion videos are first-person recordings where the view changes continuously due to the agent's movement. As they serve as the primary visual input for embodied AI agents, making egomotion video reasoning more efficient is therefore essential for real-world deployment. Recent advances in vision-language models have enabled strong multimodal reasoning capabilities, but their computational cost remains prohibitive for long, redundant video inputs. Existing token pruning methods, typically designed for third-person videos, fail to leverage the spatiotemporal continuity and motion constraints inherent in egomotion settings. To address this, we propose EgoPrune, a training-free token pruning method tailored for egomotion video reasoning. EgoPrune comprises three components: a keyframe selector adapted from EmbodiedR for temporally efficient sampling; Perspective-Aware Redundancy Filtering (PARF), which aligns visual tokens using perspective transformations and removes redundant tokens; and a Maximal Marginal Relevance (MMR)-based token selector that jointly considers visual-text relevance and intra-frame diversity. Experiments on two egomotion video benchmarks show that EgoPrune consistently outperforms prior training-free methods across various pruning ratios while significantly reducing FLOPs, memory usage, and latency. Moreover, we deploy EgoPrune on an embodied agent equipped with a Jetson Orin NX 16GB edge device, demonstrating its real-world efficiency and suitability for on-device egomotion video reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。