解决联邦跨模态检索中缺失模态问题，提升多端个性化效果
Federated Cross-Modal Retrieval with Missing Modalities via Semantic Routing and Adapter Personalization
- 用语义路由动态分配聚合权重，缓解异构更新导致的对齐漂移
- 在MS-COCO等数据集上，完整模态客户端准确率提升1.8%，缺模态客户端提升3.2%
- 适合有数据缺失或设备能力差异的分布式多模态应用
联邦跨模态检索面临严重挑战，源于客户端数据异质性，尤其是非独立同分布（non-IID）的语义分布和缺失模态。单一全局模型难以同时捕捉共享的跨模态知识与客户端特异性。本文提出RCSR框架，融合原型锚定、以检索为中心的语义路由及可选的客户端专用适配器。基于冻结的CLIP骨干网络，RCSR使用轻量级共享适配器实现全局知识迁移，同时支持高效本地个性化。原型锚定帮助单模态客户端对齐全局跨模态语义，服务器端的语义路由器根据检索一致性自适应分配聚合权重，缓解异构更新中的对齐漂移。在MS-COCO、Flickr30K等基准上的大量实验表明，RCSR持续提升全局检索精度和训练稳定性，并进一步增强客户端检索性能，尤其在模态不完整时表现更优。代码已公开于https://github.com/RezinChow/RCSR-Retrieval-Centric-Semantic-Routing。
原文摘要 · Abstract (English)
Federated cross-modal retrieval faces severe challenges from heterogeneous client data, particularly non-IID semantic distributions and missing modalities. Under such heterogeneity, a single global model is often insufficient to capture both shared cross-modal knowledge and client-specific characteristics. We propose RCSR, a personalization-friendly federated framework that integrates prototype anchoring, retrieval-centric semantic routing, and optional client-specific adapters. Built on a frozen CLIP backbone, RCSR leverages lightweight shared adapters for global knowledge transfer while supporting efficient local personalization. Prototype anchoring helps unimodal clients align with global cross-modal semantics, and a server-side semantic router adaptively assigns aggregation weights based on retrieval consistency to mitigate alignment drift during heterogeneous updates. Extensive experiments on MS-COCO, Flickr30K, and other benchmarks show that RCSR consistently improves global retrieval accuracy and training stability, while further enhancing client-level retrieval performance, especially for clients with incomplete modalities. Code is available at https://github.com/RezinChow/RCSR-Retrieval-Centric-Semantic-Routing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。