用轻量级频域处理提升事件相机行人属性识别精度
PFM-VEPAR: Prompting Foundation Models for RGB-Event Camera based Pedestrian Attribute Recognition
- 用DCT/IDCT直接提取事件数据频域特征,计算开销极低
- 通过外部记忆库与霍普菲尔德网络挖掘全局关联知识
- 适合低光、模糊场景下的行人属性识别研究者使用
基于事件的行人属性识别(PAR)利用运动线索增强低光照和运动模糊场景下的RGB相机性能,可更准确地推断年龄、情绪等属性。现有双流多模态融合方法计算开销大,且忽略上下文样本提供的有效指导。为此,本文提出事件提示器(Event Prompter)。该模块摒弃高成本的辅助主干网络,直接对事件数据应用极轻量级的离散余弦变换(DCT)和逆DCT(IDCT),以极低计算成本提取频域特征,有效增强RGB分支。此外,设计外部记忆库结合现代霍普菲尔德网络,实现关联记忆增强的表征学习,有效挖掘并利用跨样本的全局关系知识。最后,通过交叉注意力机制融合RGB与事件模态,并经前馈网络完成属性预测。在多个基准数据集上的大量实验充分验证了所提框架的有效性。代码将发布于https://github.com/Event-AHU/OpenPAR。
原文摘要 · Abstract (English)
Event-based pedestrian attribute recognition (PAR) leverages motion cues to enhance RGB cameras in low-light and motion-blur scenarios, enabling more accurate inference of attributes like age and emotion. However, existing two-stream multimodal fusion methods introduce significant computational overhead and neglect the valuable guidance from contextual samples. To address these limitations, this paper proposes an Event Prompter. Discarding the computationally expensive auxiliary backbone, this module directly applies extremely lightweight and efficient Discrete Cosine Transform (DCT) and Inverse DCT (IDCT) operations to the event data. This design extracts frequency-domain event features at a minimal computational cost, thereby effectively augmenting the RGB branch. Furthermore, an external memory bank designed to provide rich prior knowledge, combined with modern Hopfield networks, enables associative memory-augmented representation learning. This mechanism effectively mines and leverages global relational knowledge across different samples. Finally, a cross-attention mechanism fuses the RGB and event modalities, followed by feed-forward networks for attribute prediction. Extensive experiments on multiple benchmark datasets fully validate the effectiveness of the proposed RGB-Event PAR framework. The source code of this paper will be released on https://github.com/Event-AHU/OpenPAR
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。