通过语义聚类扩充放射科报告，提升视觉语言模型性能
SemEnrich: Self-Supervised Semantic Enrichment of Radiology Reports for Vision-Language Learning
- 基于报告句子语义聚类自监督扩充阳性/中性发现
- 在多个指标上平均提升5.6%-12.8%，显著改善模型表现
- 适合研究医疗视觉语言模型与数据增强的学者
医学视觉语言数据集常因规模小且偏向负面发现而受限，因临床医生通常只记录异常，可能遗漏阳性或中性发现。本文提出一种自监督数据增强方法，通过报告句子的语义聚类，在训练集中自监督地添加来自不同聚类的阳性/中性观察。该方法在监督微调中取得稳定提升：COMET、BertScore、Sentence Bleu、CheXbert-F1 和 RadGraph-F1 分别提升 5.63%、3.04%、7.40%、5.30%、7.47%。消融实验表明，性能提升源于语义聚类而非随机增强。此外，我们提出将语义聚类信息融入 GRPO 训练的奖励设计，进一步带来 2.78%、3.14%、12.80% 的平均提升。代码已开源。
原文摘要 · Abstract (English)
Medical vision-language datasets are often limited in size and biased toward negative findings, as clinicians report abnormalities mostly but might omit some positive/neutral findings because they might be considered as irrelevant to the patient's condition. We propose a self-supervised data enrichment method that leverages semantic clustering of report sentences. Then we enrich the findings in the medical reports in the training set by adding positive/neutral observations from different clusters in a self-supervised manner. Our approach yields consistent gains in supervised fine-tuning (5.63%, 3.04%, 7.40%, 5.30%, 7.47% average gains on COMET score, Bert score, Sentence Bleu, CheXbert-F1 and RadGraph-F1 scores respectively). Ablation studies confirm that improvements stem from semantic clustering rather than random augmentation. Furthermore, we introduce a way to incorporate semantic cluster information into the reward design for GRPO training, which leads to further performance gains (2.78%, 3.14%, 12.80% average gains on COMET score, Bert score and Sentence Bleu scores respectively). We share our code at https://github.com/igulluk/SemEnrich
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。