arXiv:2603.00609cs.CV2026-03中稿 · CVPR被引 2

解决多模态感知中因数据不共现导致的对齐难题,实现零共现下的高效跨模态对齐。

Linking Modality Isolation in Heterogeneous Collaborative Perception

  • 提出CodeAlign框架,通过特征-码本-特征的翻译机制实现无共现对齐。
  • 仅需8%参数量、通信负载降低1024倍,性能超越现有方法。
  • 适合自动驾驶等多设备异构协同感知场景,尤其适用于模态不共现的复杂环境。

协同感知通过多智能体间的数据交换提升整体感知能力。然而,智能体间的异质性引入领域差距,且未被充分研究的模态隔离问题进一步加剧了这一挑战:当不同模态的智能体在训练数据中从不共现时,会扩大跨模态领域差距。现有对齐方法依赖空间重叠观测的监督信号,无法处理模态隔离。为此,我们提出CodeAlign,首个无需共现的高效对齐框架,通过跨模态特征-码本-特征(FCF)转换实现平滑对齐。核心思想是利用码本显式识别表示一致性,并直接学习模态特异性特征空间间的映射,从而消除对空间对应关系的需求。码本将特征空间正则化为紧凑而富有表现力的码空间。为每种模态预设码空间后,CodeAlign学习特征到其他模态对应码的映射,并解码回目标码空间特征,实现有效对齐。实验表明,在融合三种模态时,CodeAlign仅需先前方法8%的训练参数,通信负载降低1024倍,并在OPV2V与DAIR-V2X数据集上达到当前最优感知性能。代码将开源于https://github.com/cxliu0314/CodeAlign。

原文摘要 · Abstract (English)

Collaborative perception leverages data exchange among multiple agents to enhance overall perception capabilities. However, heterogeneity across agents introduces domain gaps that hinder collaboration, and this is further exacerbated by an underexplored issue: modality isolation. It arises when multiple agents with different modalities never co-occur in any training data frame, enlarging cross-modal domain gaps. Existing alignment methods rely on supervision from spatially overlapping observations, thus fail to handle modality isolation. To address this challenge, we propose CodeAlign, the first efficient, co-occurrence-free alignment framework that smoothly aligns modalities via cross-modal feature-code-feature(FCF) translation. The key idea is to explicitly identify the representation consistency through codebook, and directly learn mappings between modality-specific feature spaces, thereby eliminating the need for spatial correspondence. Codebooks regularize feature spaces into code spaces, providing compact yet expressive representations. With a prepared code space for each modality, CodeAlign learns FCF translations that map features to the corresponding codes of other modalities, which are then decoded back into features in the target code space, enabling effective alignment. Experiments show that, when integrating three modalities, CodeAlign requires only 8% of the training parameters of prior alignment methods, reduces communication load by 1024x, and achieves state-of-the-art perception performance on both OPV2V and DAIR-V2X dataset. Code will be released on https://github.com/cxliu0314/CodeAlign.

协同感知多模态对齐自动驾驶码本机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。