arXiv:2507.03917cs.LGcs.CV2025-07IJCAI被引 11

解决多模态数据缺失与错位问题,提升数据融合质量。

Consistency-Aware Padding for Incomplete Multi-Modal Alignment Clustering Based on Self-Repellent Greedy Anchor Search

  • 通过自排斥贪心锚点搜索定位关键数据点。
  • 基于噪声对比学习实现一致性填充,提升对齐效果。
  • 适用于传感器频率不一、设备故障导致的数据错位场景。

多模态表征能有效描述真实数据的互补特征,但实际采集数据常因传感器频率不一致或设备故障而存在缺失与错位问题。现有方法仅依赖可用数据的类别级对齐,难以处理不平衡且错位的多视图数据,导致部分样本匹配不佳,影响数据融合质量。本文提出一种基于自排斥贪心锚点搜索的一致性感知填充方法(CAPIMAC),用于解决多模态数据中不平衡与错位的填充问题。具体地,设计自排斥贪心锚点搜索模块(SRGASM),结合自排斥随机游走与贪心算法,识别关键锚点以重构不完整且错位的多模态数据。随后,基于噪声对比学习构建一致性感知填充模块(CAPM),有效插值并对齐不平衡与错位数据,从而提升多模态数据融合质量。实验结果表明,该方法在多个基准数据集上优于现有基线。代码将公开于 https://github.com/Autism-mm/CAPIMAC.git。

原文摘要 · Abstract (English)

Multimodal representation is faithful and highly effective in describing real-world data samples' characteristics by describing their complementary information. However, the collected data often exhibits incomplete and misaligned characteristics due to factors such as inconsistent sensor frequencies and device malfunctions. Existing research has not effectively addressed the issue of filling missing data in scenarios where multiview data are both imbalanced and misaligned. Instead, it relies on class-level alignment of the available data. Thus, it results in some data samples not being well-matched, thereby affecting the quality of data fusion. In this paper, we propose the Consistency-Aware Padding for Incomplete Multimodal Alignment Clustering Based on Self-Repellent Greedy Anchor Search(CAPIMAC) to tackle the problem of filling imbalanced and misaligned data in multimodal datasets. Specifically, we propose a self-repellent greedy anchor search module(SRGASM), which employs a self-repellent random walk combined with a greedy algorithm to identify anchor points for re-representing incomplete and misaligned multimodal data. Subsequently, based on noise-contrastive learning, we design a consistency-aware padding module (CAPM) to effectively interpolate and align imbalanced and misaligned data, thereby improving the quality of multimodal data fusion. Experimental results demonstrate the superiority of our method over benchmark datasets. The code will be publicly released at https://github.com/Autism-mm/CAPIMAC.git.

多模态数据对齐缺失填充

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。