提出新型3D医学影像分块编码方法，提升报告生成与图像合成精度。
Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imaging
- 用因果卷积统一2D/3D训练，生成频率感知的紧凑体素令牌
- 三阶段训练使模型处理超300切片扫描，性能提升40%临床F1值
- 适合需要高精度3D医学影像生成与理解的研究者使用
三维医学影像中的视觉-语言建模近年得益于大规模CT语料库、更强架构和预训练模型的发展，实现了自动报告生成与文本条件3D图像合成等应用。然而现有方法在高分辨率长序列体积数据上表现不佳：对比学习常导致视觉编码器与临床语言错位，逐切片标记则模糊精细解剖结构，降低下游任务诊断性能。本文提出BTB3D（Better Tokens for Better 3D），一种因果卷积编码器-解码器架构，统一2D与3D训练与推理，生成紧凑且频率感知的体素令牌。通过三阶段训练流程：(i) 局部重建，(ii) 重叠窗口拼贴，(iii) 长上下文解码器优化，模型在仅用短切片片段训练的情况下，即可泛化至超过300切片的扫描，无额外内存开销。BTB3D在两项关键任务上刷新基准：报告生成方面，相比CT2Rep、CT-CHAT和Merlin，BLEU分数提升，临床F1值提高40%；文本到CT合成方面，相比GenerateCT和MedSyn，FID降低75%，FVD减半，生成512×512×241的解剖一致体数据。结果表明，精确的三维标记机制比单纯增大语言模型规模更为关键。
原文摘要 · Abstract (English)
Recent progress in vision-language modeling for 3D medical imaging has been fueled by large-scale computed tomography (CT) corpora with paired free-text reports, stronger architectures, and powerful pretrained models. This has enabled applications such as automated report generation and text-conditioned 3D image synthesis. Yet, current approaches struggle with high-resolution, long-sequence volumes: contrastive pretraining often yields vision encoders that are misaligned with clinical language, and slice-wise tokenization blurs fine anatomy, reducing diagnostic performance on downstream tasks. We introduce BTB3D (Better Tokens for Better 3D), a causal convolutional encoder-decoder that unifies 2D and 3D training and inference while producing compact, frequency-aware volumetric tokens. A three-stage training curriculum enables (i) local reconstruction, (ii) overlapping-window tiling, and (iii) long-context decoder refinement, during which the model learns from short slice excerpts yet generalizes to scans exceeding 300 slices without additional memory overhead. BTB3D sets a new state-of-the-art on two key tasks: it improves BLEU scores and increases clinical F1 by 40% over CT2Rep, CT-CHAT, and Merlin for report generation; and it reduces FID by 75% and halves FVD compared to GenerateCT and MedSyn for text-to-CT synthesis, producing anatomically consistent 512*512*241 volumes. These results confirm that precise three-dimensional tokenization, rather than larger language backbones alone, is essential for scalable vision-language modeling in 3D medical imaging. The codebase is available at: https://github.com/ibrahimethemhamamci/BTB3D
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。