针对医学影像报告生成难题，提出自适应注意力与三路LSTM结合的新模型。
Image-to-Text for Medical Reports Using Adaptive Co-Attention and Triple-LSTM Module
- 用自适应注意力机制增强相似医学图像的区分能力
- 三路LSTM模块提升生成报告的语义准确性与连贯性
- 在三个公开数据集上优于主流模型，尤其适合临床辅助场景
医学报告生成需要专业领域知识，通用大模型常难以准确捕捉。同时，医学数据固有的重复性和相似性使模型难以提取有效特征，易产生过拟合。为此，本文提出一种多模态模型——协同注意力三路LSTM网络（CA-TriNet），结合Transformer架构与多路LSTM网络。其协同注意力模块将视觉Transformer与文本Transformer联动，更好区分外观相似的医学图像，并引入自适应权重算子放大微小差异的图像标签。此外，三路LSTM模块通过聚焦图像中的关键对象优化生成句子。在三个公开数据集上的大量实验表明，CA-TriNet在综合性能上超越当前最优模型，甚至在部分指标上优于预训练大语言模型。
原文摘要 · Abstract (English)
Medical report generation requires specialized expertise that general large models often fail to accurately capture. Moreover, the inherent repetition and similarity in medical data make it difficult for models to extract meaningful features, resulting in a tendency to overfit. So in this paper, we propose a multimodal model, Co-Attention Triple-LSTM Network (CA-TriNet), a deep learning model that combines transformer architectures with a Multi-LSTM network. Its Co-Attention module synergistically links a vision transformer with a text transformer to better differentiate medical images with similarities, augmented by an adaptive weight operator to catch and amplify image labels with minor similarities. Furthermore, its Triple-LSTM module refines generated sentences using targeted image objects. Extensive evaluations over three public datasets have demonstrated that CA-TriNet outperforms state-of-the-art models in terms of comprehensive ability, even pre-trained large language models on some metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。