解决多模态数据缺失下的精准检索问题，无需生成模型即可应对任意模态缺失。
Any2Any: Incomplete Multimodal Retrieval with Conformal Prediction
- 基于交叉模态编码器计算相似度，结合置信区间校准实现跨模态对齐。
- 在KITTI数据集上达Recall@5为35%，与完整模态基线持平。
- 适用于文本-激光雷达、文本-时序等任意不完整模态组合，适合机器人感知场景。
自主代理通过融合视觉、音频和激光雷达等多模态输入来感知与理解环境，支持如机器人定位中的场景检索任务。然而，当前系统在传感器故障或信息不可用导致部分数据缺失（如无声视频、无RGB的激光雷达扫描）时表现不佳。本文提出Any2Any——一种新型检索框架，可处理查询与参考样本均存在任意模态缺失的情形。不同于以往仅限于两类模态补全的方法，Any2Any无需训练生成模型，即可处理任意数量的缺失模态。它通过交叉模态编码器计算成对相似度，并采用两阶段置信区间校准（conformal prediction）进行对齐。该方法在文本-激光雷达、文本-时序等跨模态数据集上实现有效检索，在KITTI数据集上达到Recall@5为35%，与拥有完整模态的基线模型性能相当。
原文摘要 · Abstract (English)
Autonomous agents perceive and interpret their surroundings by integrating multimodal inputs, such as vision, audio, and LiDAR. These perceptual modalities support retrieval tasks, such as place recognition in robotics. However, current multimodal retrieval systems encounter difficulties when parts of the data are missing due to sensor failures or inaccessibility, such as silent videos or LiDAR scans lacking RGB information. We propose Any2Any-a novel retrieval framework that addresses scenarios where both query and reference instances have incomplete modalities. Unlike previous methods limited to the imputation of two modalities, Any2Any handles any number of modalities without training generative models. It calculates pairwise similarities with cross-modal encoders and employs a two-stage calibration process with conformal prediction to align the similarities. Any2Any enables effective retrieval across multimodal datasets, e.g., text-LiDAR and text-time series. It achieves a Recall@5 of 35% on the KITTI dataset, which is on par with baseline models with complete modalities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。