提出轻量级病理图像多模态模型，高效压缩与筛选关键视觉信息。
LoC-Path: Learning to Compress for Pathology Multimodal Large Language Models
- 先压缩后融合：用稀疏合并与预训练重采样降低视觉特征冗余。
- 仅用少量相关潜变量，推理延迟和内存消耗显著下降。
- 适合资源受限场景下的病理诊断模型部署，兼顾性能与效率。
全切片图像（WSI）多模态大语言模型（MLLM）构建与部署困难，因千兆像素切片产生数千个视觉标记，但仅有少量区域具有诊断意义。现有模型通常依赖重型切片编码器和长视觉前缀，导致端到端训练与部署在计算资源有限时成本高昂。本文重新审视该问题，发现WSI切片特征在全局与局部尺度均高度冗余，而任务相关证据稀疏且依赖查询。为此提出LoC-Path，一种资源高效的切片级MLLM，其核心是压缩后再融合：使用稀疏标记合并器（STM）和MAE预训练重采样器替代昂贵的切片级编码，生成紧凑潜界面；再通过标记重要性评分器（TIS）筛选最相关潜变量，并用跨注意力路由适配器（CARA）将其融合至少量LLM解码层。实验表明，该设计显著降低多模态微调成本与推理时延迟/内存占用，在有限资源下仍保持与先前模型相当的性能，使端到端开发与部署更可行。
原文摘要 · Abstract (English)
Whole Slide Image (WSI) MLLMs are difficult to build and deploy because gigapixel slides induce thousands of visual tokens, while only a small fraction of regions is diagnostically relevant. Existing slide-level pathology MLLMs typically combine heavy slide-level encoders with long visual prefixes, making end-to-end slide-level development and deployment expensive under limited computational resources. We revisit this regime and show that WSI tile features are highly redundant at both global and local scales, while task-relevant evidence is sparse and query-dependent. We therefore introduce LoC-Path, a resource-efficient slide-level MLLM that compresses before fusion. LoC-Path uses a Sparse Token Merger (STM) and an MAE-pretrained resampler to replace expensive slide-level encoding with a compact latent interface, then uses a Token Importance Scorer (TIS) to select the most relevant latents and a Cross-Attention Routing Adapter (CARA) to fuse them into a few LLM decoder layers. This design lowers both multimodal tuning cost and inference-time latency/memory by avoiding heavy slide-level encoding and long visual prefixes. Extensive experiments show that LoC-Path remains competitive with prior slide-level MLLMs while making end-to-end development and deployment more practical under limited computational resources.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。