用共享代码本实现无配对医学影像跨模态知识迁移
Shared Semantic Codebook Distillation for Unpaired Cross-Modal Medical Classification
- 通过共享离散代码本比较不同模态表示，无需配对样本
- OCT转眼底图像分类提升64.5→70.2，CT转胸片分类73.8→76.3
- 适合资源受限场景下医学影像模型轻量化部署
跨模态知识蒸馏可将强但昂贵的教师模态诊断知识迁移到更便宜且易部署的学生模态。但在医学图像分析中，两模态常为无配对数据：来自不同患者队列，特征空间几何不兼容。这使得实例级蒸馏无效，直接特征匹配不可靠。为此，我们提出共享语义代码本蒸馏（SSCD），通过共享离散代码本比较教师与学生表示。每张图像被表示为通用、模态无关词汇上的分布，知识通过跨模态对齐这些分布实现，支持全局与类别条件对齐，无需配对样本或可直接比较的原始特征。代码本通过指数移动平均在线演化，并通过熵正则化和死代码重启保持多样性。推理时，所有教师端及代码本模块均被移除，仅保留学生编码器与分类器。在两个异构无配对设置下——OCT-to-fundus视网膜疾病分类与CT-to-chest-X-ray肺炎分类——SSCD使学生模型的宏平均F1分别从64.5提升至70.2，从73.8提升至76.3，优于所有评估基线。代码与预训练模型见https://github.com/DillanImans/SSCD-unpaired-distillation
原文摘要 · Abstract (English)
Cross-modal knowledge distillation can transfer diagnostic knowledge from a strong but costly teacher modality to a cheaper and more deployable student modality. In medical image analysis, however, the two modalities are often unpaired: they are collected from different patient cohorts and occupy geometrically incompatible feature spaces. This makes instance-level distillation invalid and direct feature matching unreliable. To address these challenges, we propose Shared Semantic Codebook Distillation (SSCD), which compares teacher and student representations through a shared discrete codebook. Each image is represented as a distribution over a common, modality-agnostic vocabulary, and knowledge is transferred by aligning these distributions across modalities, both globally and class-conditionally, without requiring paired samples or directly comparable raw features. The codebook is evolved online by exponential moving average and kept diverse through entropy regularization and dead-code restart. At inference, all teacher-side and codebook modules are discarded, leaving only the student encoder and classifier. On two heterogeneous unpaired settings, OCT-to-fundus retinal disease classification and CT-to-chest-X-ray pneumonia classification, SSCD improves the student from 64.5 to 70.2 macro-F1 and from 73.8 to 76.3 macro-F1, respectively, outperforming all evaluated distillation baselines on both settings. Code and pretrained models are available at https://github.com/DillanImans/SSCD-unpaired-distillation
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。