用事件相机辅助训练更鲁棒的单模态图像模型,提升跨域泛化能力。
PEPR: Privileged Event-based Predictive Regularization for Domain Generalization
- 将事件相机信息作为训练期特权,通过预测事件特征增强图像模型鲁棒性
- 在日间到夜间等跨域场景下,检测与分割性能显著优于对齐类方法
- 适合需要高鲁棒性的自动驾驶、机器人视觉等实际部署场景
面向视觉感知的深度神经网络极易受域偏移影响,严重制约其在真实场景中的应用。为应对这一挑战,本文提出一种基于学习特权信息(LUPI)范式的跨模态框架,旨在训练一个鲁棒的单模态RGB模型。利用事件相机作为仅在训练阶段可用的特权信息源,其与RGB流具有互补特性:RGB语义丰富但域依赖性强,事件数据稀疏却更具域不变性。直接进行跨模态特征对齐会迫使RGB编码器模仿稀疏的事件表示,导致语义细节丢失。为此,本文提出特权事件预测正则化(PEPR),将LUPI重构为共享潜在空间中的预测问题。不强制对齐,而是训练RGB编码器通过预测事件潜在特征来提取鲁棒性,从而保留语义丰富性。最终得到的独立RGB模型在日间到夜间等域转移场景中表现优异,显著超越基于对齐的基线方法,在目标检测与语义分割任务上均实现持续提升。
原文摘要 · Abstract (English)
Deep neural networks for visual perception are highly susceptible to domain shift, which poses a critical challenge for real-world deployment under conditions that differ from the training data. To address this domain generalization challenge, we propose a cross-modal framework under the learning using privileged information (LUPI) paradigm for training a robust, single-modality RGB model. We leverage event cameras as a source of privileged information, available only during training. The two modalities exhibit complementary characteristics: the RGB stream is semantically dense but domain-dependent, whereas the event stream is sparse yet more domain-invariant. Direct feature alignment between them is therefore suboptimal, as it forces the RGB encoder to mimic the sparse event representation, thereby losing semantic detail. To overcome this, we introduce Privileged Event-based Predictive Regularization (PEPR), which reframes LUPI as a predictive problem in a shared latent space. Instead of enforcing direct cross-modal alignment, we train the RGB encoder with PEPR to predict event-based latent features, distilling robustness without sacrificing semantic richness. The resulting standalone RGB model consistently improves robustness to day-to-night and other domain shifts, outperforming alignment-based baselines across object detection and semantic segmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。