融合OCT与眼底照相提升视力缺损评估,但传统方法因数据失衡失效,新模型有效解决此问题。
Re-M3Dr: Rebalanced MultiModal Mean Deviation Regression

- 通过自适应边界对比学习增强单模态特征,再用梯度调制稳定多模态优化
- 在临床数据上平均降低29%的均方误差,显著优于现有方法
- 适合眼科医学影像分析、多模态学习中的不平衡问题研究者
视域缺损的均值偏差(MD)是眼科评估的重要指标。以往研究仅基于光学相干断层扫描(OCT)预测MD,而结合眼底照相(FP)可提供互补信息,理论上应提升性能。然而,我们的实验发现,多模态融合反而劣于单模态模型。深入分析表明,根本原因在于数据分布失衡与模态学习冲突的耦合,导致优化过程不稳定。为此,我们提出平衡式多模态均值偏差回归(Re-M3Dr)框架:首先通过基于自适应边界的监督对比学习增强单模态表征;随后采用尖锐感知梯度调制稳定联合优化。在公开与私有临床数据集上的实验显示,该方法相较当前最优多模态学习方法平均降低29%的均方误差,验证了其有效性。代码见附录。
原文摘要 · Abstract (English)
Mean Deviation (MD) is a critical metric for assessing visual field loss in ophthalmology. While previous work has focused solely on predicting MD from Optical Coherence Tomography (OCT), it is intuitive to assume that combining OCT with another imaging of fundus photography (FP) could improve performance, as two ophthalmic medical imaging provide complementary information. This is particularly expected when sophisticated multi-objective optimization is applied, as documented in common multimodal classification. Surprisingly, our investigations reveal that multimodal fusion in this medical imaging scenario performs worse than unimodal model. Through detailed analysis, we identify the root cause as a coupled imbalance between data distribution and modality learning conflict. This imbalance distorts the optimization landscape, leading to unstable training. To address this challenge, we propose the method of Rebalanced MultiModal Mean Deviation Regression (Re-M3Dr), a novel multimodal regression framework. We enhance unimodal representation through adaptive margin based supervised contrastive learning. Then, our framework stabilizes the joint optimization with the sharpness-aware gradient modulation. Experimental results on both public and private clinical datasets show average 29\% reduction in MSE compared to SOTA multimodal learning methods, demonstrating the superiority of Re-M3Dr. The code is available in the supplementary materials.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。