用事件数据驱动,让低帧率相机实现任意时间的高帧率分割。
LiFR-Seg: Anytime High-Frame-Rate Segmentation via Event-Guided Propagation
- 基于事件流和单张图像,实时预测任意时刻的语义分割结果。
- 在DSEC数据集上达到73.82% mIoU,接近高帧率上限(仅差0.09%)。
- 适合需要低延迟、高精度动态场景感知的自动驾驶与机器人系统。
动态环境中密集语义分割受限于标准摄像头的低帧率(LFR),导致帧间存在显著感知空白。为此,我们提出「任意时间帧间分割」:仅使用一张过去的RGB帧和异步事件数据流,在任意时间点预测分割结果。核心挑战在于如何利用稀疏且常含噪声的事件数据生成运动场,稳健地传播稠密语义特征,并在高度动态场景中抑制特征退化。我们提出LiFR-Seg框架,直接通过时间传播深层语义特征。方法核心为不确定性感知的变形过程,由事件驱动的运动场及其学习到的显式置信度引导;同时引入时序记忆注意力模块,确保动态场景中的分割一致性。我们在DSEC数据集及自建的高频合成基准SHF-DSEC上验证该方法。令人瞩目的是,我们的低帧率系统在DSEC上取得73.82% mIoU,与拥有目标帧完整信息的高帧率上限相比,差异仅为0.09%,统计上无显著区别。本工作建立了一种高效、鲁棒的高帧率感知新范式,仅依赖低帧率硬件即可实现。
原文摘要 · Abstract (English)
Dense semantic segmentation in dynamic environments is fundamentally limited by the low-frame-rate (LFR) nature of standard cameras, which creates critical perceptual gaps between frames. To solve this, we introduce Anytime Interframe Semantic Segmentation: a new task for predicting segmentation at any arbitrary time using only a single past RGB frame and a stream of asynchronous event data. This task presents a core challenge: how to robustly propagate dense semantic features using a motion field derived from sparse and often noisy event data, all while mitigating feature degradation in highly dynamic scenes. We propose LiFR-Seg, a novel framework that directly addresses these challenges by propagating deep semantic features through time. The core of our method is an uncertainty-aware warping process, guided by an event-driven motion field and its learned, explicit confidence. A temporal memory attention module further ensures coherence in dynamic scenarios. We validate our method on the DSEC dataset and a new high-frequency synthetic benchmark (SHF-DSEC) we contribute. Remarkably, our LFR system achieves performance (73.82% mIoU on DSEC) that is statistically indistinguishable from an HFR upper-bound (within 0.09%) that has full access to the target frame. This work presents a new, efficient paradigm for achieving robust, high-frame-rate perception with low-frame-rate hardware. Project Page: https://candy-crusher.github.io/LiFR_Seg_Proj/#; Code: https://github.com/Candy-Crusher/LiFR-Seg.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。