通过压缩病理切片的视觉令牌，实现高效问答。
Efficient Whole Slide Pathology VQA via Token Compression
- 引入可训练的压缩令牌，聚合多模态信息
- 在10种肿瘤亚型上准确率更高，资源消耗大幅降低
- 适合需要高效病理图像问答的研究者使用
病理全切片图像（WSI）可达10,000×10,000像素，对多模态大语言模型（MLLM）带来长上下文和高计算需求挑战。以往方法多基于CLIP与多实例学习进行局部或整体分类，缺乏生成式问答能力；近期基于MLLM的方法直接输入数千个补丁令牌，导致资源消耗过大。为此，我们提出首个通过令牌压缩实现WSI VQA的MLLM架构——TCP-LLaVA。该模型引入一组可训练压缩令牌，通过模态压缩模块聚合视觉与文本信息，借鉴BERT中的[CLS]机制。仅将压缩后的令牌输入LLM生成答案，显著减少输入长度与计算开销。在十种TCGA肿瘤亚型上的实验表明，TCP-LLaVA在VQA准确率上优于现有基线，同时训练资源消耗大幅降低。
原文摘要 · Abstract (English)
Whole-slide images (WSIs) in pathology can reach up to 10,000 x 10,000 pixels, posing significant challenges for multimodal large language model (MLLM) due to long context length and high computational demands. Previous methods typically focus on patch-level analysis or slide-level classification using CLIP-based models with multi-instance learning, but they lack the generative capabilities needed for visual question answering (VQA). More recent MLLM-based approaches address VQA by feeding thousands of patch tokens directly into the language model, which leads to excessive resource consumption. To address these limitations, we propose Token Compression Pathology LLaVA (TCP-LLaVA), the first MLLM architecture to perform WSI VQA via token compression. TCP-LLaVA introduces a set of trainable compression tokens that aggregate visual and textual information through a modality compression module, inspired by the [CLS] token mechanism in BERT. Only the compressed tokens are forwarded to the LLM for answer generation, significantly reducing input length and computational cost. Experiments on ten TCGA tumor subtypes show that TCP-LLaVA outperforms existing MLLM baselines in VQA accuracy while reducing training resource consumption by a substantial margin.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。