arXiv:2510.02789cs.CVcs.AI2025-10

让医学影像检测的查询向量对齐模态特征,提升跨模态检测效果。

Align Your Query: Representation Alignment for Multimodality Medical Object Detection

  • 用文本生成的模态标记注入检测查询,实现跨模态特征对齐。
  • 在多模态数据上提升检测平均精度(AP),且不增加计算开销。
  • 无需修改模型结构,适合实际医疗系统快速部署。

医学物体检测在混合使用多种成像模态(如X光、CT、MRI)的数据训练时,因统计差异和表示空间分离而表现下降。为解决此问题,本文提出基于表示对齐的方法,聚焦于DETR类检测器中的对象查询表示。首先定义轻量级的模态标记:由文本生成的嵌入向量,编码影像模态信息,无需额外标注。通过多模态上下文注意力(MoCA)机制将模态标记融入检测流程,利用自注意力融合对象查询表示,实现模态上下文在查询集内的传播。该方法保持DETR架构不变,仅引入极低延迟,并将模态信息注入查询中。进一步提出QueryREPA预训练阶段,使用模态均衡批次与特定对比目标对齐查询表示与模态标记。整体方法生成具有模态感知性且类别忠实的查询,可有效迁移到下游任务。在联合训练多种模态数据下,该方法持续提升平均精度(AP),几乎无额外开销,且无需修改网络结构,为构建鲁棒的多模态医学物体检测提供实用路径。

原文摘要 · Abstract (English)

Medical object detection suffers when a single detector is trained on mixed medical modalities (e.g., CXR, CT, MRI) due to heterogeneous statistics and disjoint representation spaces. To address this challenge, we turn to representation alignment, an approach that has proven effective for bringing features from different sources into a shared space. Specifically, we target the representations of DETR-style object queries and propose a simple, detector-agnostic framework to align them with modality context. First, we define modality tokens: compact, text-derived embeddings encoding imaging modality that are lightweight and require no extra annotations. We integrate the modality tokens into the detection process via Multimodality Context Attention (MoCA), mixing object-query representations via self-attention to propagate modality context within the query set. This preserves DETR-style architectures and adds negligible latency while injecting modality cues into object queries. We further introduce QueryREPA, a short pretraining stage that aligns query representations to their modality tokens using a task-specific contrastive objective with modality-balanced batches. Together, MoCA and QueryREPA produce modality-aware, class-faithful queries that transfer effectively to downstream training. Across diverse modalities trained altogether, the proposed approach consistently improves AP with minimal overhead and no architectural modifications, offering a practical path toward robust multimodality medical object detection.

多模态医学检测表示对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。