arXiv:2411.15539cs.CVcs.AI2024-11被引 42

通过区域引导定位与对齐,提升肺部CT报告生成的精准性与可解释性。

Large Language Model with Region-guided Referring and Grounding for CT Report Generation

  • 用分割掩码捕捉局部区域特征,结合全局信息生成上下文连贯报告。
  • 在两个胸部CT数据集上,报告生成准确率和临床指标均优于现有方法。
  • 适合医学影像分析、放射科辅助诊断等需要高可解释性的场景。

计算机断层扫描(CT)报告生成对辅助放射科医生解读体积数据至关重要,但耗时且费力。现有方法主要依赖整体体积的全局特征,难以聚焦特定区域,易遗漏异常。为此,我们提出首个区域引导指代与定位框架Reg2RG,通过关注体积内的解剖区域来提升诊断性能。具体地,利用通用分割模块生成的掩码提取各指代区域的局部特征,并提出局部特征解耦(LFD)策略,在极低计算开销下保留高分辨率细节。局部特征与全局特征融合以捕捉区域间关联。此外,设计新型区域-报告对齐(RRA)训练策略,借助指代区域识别引导生成区域特异性报告,增强模型指代与定位能力并提升报告可解释性。进一步采用大语言模型(LLM)作为语言解码器,从整合的视觉特征生成报告,实现区域级理解。在两个大规模胸部CT-报告数据集上的大量实验表明,本方法在自然语言生成与临床效能指标上均优于多个先进方法,同时保持良好可解释性。代码已开源:https://github.com/zhi-xuan-chen/Reg2RG。

原文摘要 · Abstract (English)

Computed tomography (CT) report generation is crucial to assist radiologists in interpreting CT volumes, which can be time-consuming and labor-intensive. Existing methods primarily only consider the global features of the entire volume, making it struggle to focus on specific regions and potentially missing abnormalities. To address this issue, we propose Reg2RG, the first region-guided referring and grounding framework for CT report generation, which enhances diagnostic performance by focusing on anatomical regions within the volume. Specifically, we utilize masks from a universal segmentation module to capture local features for each referring region. A local feature decoupling (LFD) strategy is proposed to preserve the local high-resolution details with little computational overhead. Then the local features are integrated with global features to capture inter-regional relationships within a cohesive context. Moreover, we propose a novel region-report alignment (RRA) training strategy. It leverages the recognition of referring regions to guide the generation of region-specific reports, enhancing the model's referring and grounding capabilities while also improving the report's interpretability. A large language model (LLM) is further employed as the language decoder to generate reports from integrated visual features, facilitating region-level comprehension. Extensive experiments on two large-scale chest CT-report datasets demonstrate the superiority of our method, which outperforms several state-of-the-art methods in terms of both natural language generation and clinical efficacy metrics while preserving promising interpretability. The code is available at https://github.com/zhi-xuan-chen/Reg2RG.

CT报告生成区域引导大语言模型医学影像

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。