提出轻量级令牌选择模块，高效处理事件数据中的冗余信息。
PSTTS: A Plug-and-Play Token Selector for Efficient Event-based Spatio-temporal Representation Learning
- 基于原始事件数据的时空分布特性，分两阶段剔除冗余令牌
- 在多个数据集上降低29%-43.6%计算量，帧率提升21.6%-41.3%
- 无需额外参数，可直接接入主流事件模型，适合实时系统部署
主流事件时空表征学习方法通常将事件流转换为事件帧序列，虽表现优异，却忽视了事件帧序列中固有的高空间稀疏性与帧间运动冗余，导致显著计算开销。现有针对RGB视频的令牌稀疏化方法依赖不可靠的中间令牌表示，且忽略事件噪声影响，难以直接用于事件数据。本文提出渐进式时空令牌选择（PSTTS），一个无额外参数的即插即用模块。PSTTS利用原始事件数据中的时空分布特性，有效识别并剔除时空冗余令牌，在精度与效率间取得最优平衡。具体包括空间令牌净化与时间令牌选择两阶段：前者通过评估每帧内事件的时空一致性，剔除噪声与非事件区域，避免干扰后续时序冗余判断；后者基于相邻帧间运动模式相似性，精准定位并移除冗余时序信息。PSTTS应用于UniformerV2、VideoSwin、EVMamba和ExACT四个代表性主干网络，在HARDVS、DailyDVS-200和SeACT数据集上验证。实验表明，PSTTS在保持任务精度的同时，显著提升效率：在DailyDVS-200上减少29%-43.6%的FLOPs，帧率提升21.6%-41.3%。代码将公开。
原文摘要 · Abstract (English)
Mainstream event-based spatio-temporal representation learning methods typically process event streams by converting them into sequences of event frames, achieving remarkable performance. However, they neglect the high spatial sparsity and inter-frame motion redundancy inherent in event frame sequences, leading to significant computational overhead. Existing token sparsification methods for RGB videos rely on unreliable intermediate token representations and neglect the influence of event noise, making them ineffective for direct application to event data. In this paper, we propose Progressive Spatio-Temporal Token Selection (PSTTS), a Plug-and-Play module for event data without introducing any additional parameters. PSTTS exploits the spatio-temporal distribution characteristics embedded in raw event data to effectively identify and discard spatio-temporal redundant tokens, achieving an optimal trade-off between accuracy and efficiency. Specifically, PSTTS consists of two stages, Spatial Token Purification and Temporal Token Selection. Spatial Token Purification discards noise and non-event regions by assessing the spatio-temporal consistency of events within each event frame to prevent interference with subsequent temporal redundancy evaluation. Temporal Token Selection evaluates the motion pattern similarity between adjacent event frames, precisely identifying and removing redundant temporal information. We apply PSTTS to four representative backbones UniformerV2, VideoSwin, EVMamba, and ExACT on the HARDVS, DailyDVS-200, and SeACT datasets. Experimental results demonstrate that PSTTS achieves significant efficiency improvements. Specifically, PSTTS reduces FLOPs by 29-43.6% and increases FPS by 21.6-41.3% on the DailyDVS-200 dataset, while maintaining task accuracy. Our code will be available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。