让医学视觉语言模型用上长文本，提升检索与分类性能
No Tokens Wasted: Leveraging Long Context in Biomedical Vision-Language Models
- 扩展文本编码器上下文至512词元，减少长摘要截断
- 在长图文检索上召回率最高提升30%，分类准确率提高2%
- 适合需要精准理解医学文献的科研与临床场景
将视觉语言模型（VLMs）预训练时的文本窗口长度通常限制在77词元以下，导致大量医学文献中的长摘要被截断。然而，从大规模开源文献中发现，多数医学摘要远超77词元。为此，我们研究了在长格式医学摘要上进行预训练的影响，通过扩展文本编码器的上下文长度来利用更完整的监督信号。结果表明，更长的上下文能显著提升检索与分类表现。基于此，我们构建了包含100万张图像-摘要对的BIOMEDICA-LongCAP数据集，其描述源自全文文章，提供更长且额外的文本监督。在此基础上，我们训练了支持最长512词元的BMC-LongCLIP模型。该模型将上下文容量扩大6.6倍，将词元浪费从55%降至2.2%。在长摘要检索任务中，BMC-LongCLIP的Recall@1最高提升30个百分点，分类任务平均提升2%，同时收敛速度更快。结果表明，长上下文建模是推动医学视觉语言模型发展的有效方向。
原文摘要 · Abstract (English)
Embedding vision-language models (VLMs) are typically pretrained with short text windows (<77 tokens), which forces the truncation of long-format captions. Yet, the distribution of biomedical captions from large-scale open source literature reveals that a huge portion of captions far exceed 77 tokens. To this end, we investigate the impact of pretraining on long-format biomedical captions by extending the context length of text encoders in VLMs. We find that longer context (thus, enabling additional supervision provided in long-format captions) correlates with better retrieval and classification performance. Given this finding, we introduce BIOMEDICA-LongCAP, a dataset of 1M image-caption pairs enriched with context-aware descriptions from full-text articles, providing longer and additional textual supervision. Using BIOMEDICA-LongCAP, we train BMC-LongCLIP, a long-context biomedical VLM with a text encoder supporting windows of up to 512 tokens. Our model extends context capacity by 6.6x, reducing token waste from 55% to just 2.2%. On long-caption retrieval benchmarks, BMC-LongCLIP achieves up to +30% absolute gains in Recall@1 and +2% average improvements in classification, while also converging faster than short-context. Our results demonstrate that long-context modeling is a promising direction for advancing biomedical VLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。