让模型自己学着挑关键图像块,大幅压缩计算量却不丢重要病理信息。
Learnable Token Sparsification for Efficient Gigapixel Whole Slide Image Reasoning

- 设计可训练的稀疏化机制,让模型自主选择最具诊断价值的图像块。
- 仅保留32个关键令牌(原长度0.78%),在SlideBench上达73.32%准确率。
- 适合需要高效处理高分辨率病理切片的医学AI研究者使用。
在视觉语言模型中处理千兆像素级全切片图像时,视觉令牌数量过多带来巨大计算负担。现有方法多依赖空间下采样或无训练的启发式剪枝,常因忽略不规则分布的细微病理特征而失效。为此,本文将切片令牌压缩重构为可训练的稀疏化问题,提出解耦路由架构。为使梯度通过不可微剪枝操作传播,引入SparseLearn模块:采用保持方差的噪声门与可微软Top-K算子调控每块信息流,并通过对角注意力去噪器恢复被扰动表征且不泄露空间信息。推理时,SparseLearn模块完全移除,训练好的评分器以确定性硬Top-K保留最高得分的32个令牌,无额外开销。该框架将视觉序列压缩至仅32个令牌(原长0.78%),在SlideBench(TCGA)上实现73.32%整体准确率,持续优于基于采样的基线和通用视觉语言模型,且在SlideBench(BCNB)和WSI VQA*上展现强零样本泛化能力。该工作解决了视觉上下文瓶颈问题,有效防止稀疏诊断证据被稀释,为端到端千兆像素全切片图像推理提供高效范式。
原文摘要 · Abstract (English)
The processing of gigapixel whole slide images within vision language models faces a major difficulty due to an excessive number of visual tokens. Existing solutions typically rely on spatial downsampling or heuristic pruning strategies that operate without training, and these methods often discard subtle but clinically meaningful patterns because pathological evidence is scattered irregularly across the tissue. To overcome this limitation, we reformulate token reduction in whole slide images as a trainable sparsification problem, allowing the model to learn an optimal selection strategy instead of following fixed heuristics. We propose a decoupled routing architecture. To enable gradient propagation through the nondifferentiable pruning operation during training, we introduce a component called SparseLearn. This component uses a variance-preserving noise gate that regulates the information flow of each patch via a differentiable Soft Top-K operator, together with a diagonal attention denoiser that recovers perturbed representations without leaking spatial information. At inference time, the SparseLearn module is entirely discarded, and the trained scorer applies a deterministic Hard Top-K operator to keep only the highest scoring 32 tokens, incurring no extra computation. By compressing the visual sequence down to a sparse set of just 32 tokens, which represents as little as 0.78% of the original length, our framework achieves 73.32% overall accuracy on SlideBench (TCGA), consistently surpassing sampling-based baselines and general-purpose vision language models. It also demonstrates strong zero shot generalization on SlideBench (BCNB) and WSI VQA*. By resolving the visual context bottleneck and preventing the dilution of sparse diagnostic evidence, this work provides a highly efficient paradigm for end to end gigapixel whole slide image reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。