arXiv:2607.09143cs.CV2026-07

提出首个专为密集RGB-事件融合设计的统一模型,解决时空错位与跨谱混叠问题。

Weaving Light and Time: Unified Harmonic-Geometric Representation Learning for Dense RGB-Event Parsing

论文配图:Weaving Light and Time: Unified Harmonic-Geometric Representation Learning for Dense RGB-Event Parsing
图 1 · 摘自论文原文
  • 在每层编码器中嵌入几何对齐、频域纹理迁移和事件驱动注意力模块
  • 在DELIVER/DDD17/DSEC上均达新SOTA,推理延迟更低
  • 支持任意事件格式,适合实时多模态感知场景

融合标准RGB帧与异步事件流已成为退化环境下鲁棒感知的主流范式。尽管统一骨干网络在多模态视觉中逐渐兴起,但将其适配至RGB-事件领域仍面临根本挑战。现有架构或采用双分支解耦编码器导致计算量翻倍,或使用通用统一设计,无法解决密集强度网格与稀疏运动尖峰之间的隐式几何偏移及跨谱混叠问题。为此,本文提出Evita,首个专为密集RGB-事件解析设计的统一骨干网络。为实现深层模态协同,Evita在每一编码层显式嵌入一组内在共学习模块:几何偏移校正用于自适应空间对齐,谐波谱共振实现复频域专属纹理迁移,瞬态全局路由支持事件驱动的非对称注意力。为增强特征提取鲁棒性并解耦表示对特定事件编码的依赖,构建了N-ImageNetV2数据集与随机事件表示混合预训练协议,使网络可无缝适配下游任务中的任意事件格式。在DELIVER、DDD17与DSEC基准上的广泛评估表明,Evita在多项指标上建立新SOTA,同时实现更优的精度-延迟权衡。代码已公开于https://github.com/chaineypung/Evita。

原文摘要 · Abstract (English)

Fusing standard RGB frames with asynchronous event streams has emerged as a definitive paradigm for robust perception in degraded environments. Although unified backbones have recently gained traction in multi-modal vision, adapting them to the RGB-Event domain remains fundamentally challenging. Existing architectures either resort to decoupled dual encoders that double computational overhead, or adopt generic unified designs that fail to resolve implicit geometric parallax and cross-spectral aliasing under the extreme representational divide between dense intensity grids and sparse kinematic spikes. To transcend these bottlenecks, we present Evita, the first unified backbone specifically engineered for dedicated dense RGB-Event parsing. To achieve profound modal synergy, Evita explicitly embeds a suite of intrinsic co-learning modules directly into every encoder layer. Specifically, it features Geometric Parallax Rectification for adaptive spatial alignment, Harmonic Spectral Resonance for texture transfer exclusively in the complex frequency domain, and Transient Global Routing for event-driven asymmetric attention. To guarantee robust feature extraction against spatial misalignments and decouple representations from specific event encodings, we construct N-ImageNetV2 alongside a stochastic event representation mixing pretraining protocol, empowering the network to seamlessly accommodate arbitrary event formats in downstream tasks. Extensive evaluations across the DELIVER, DDD17, and DSEC benchmarks confirm that Evita establishes new state-of-the-art metrics while delivering a superior accuracy-latency trade-off for real-time multimodal perception.The code are publicly available at: https://github.com/chaineypung/Evita.

RGB-事件融合统一架构实时感知多模态学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。