用视觉语言模型为单颗牙图像生成精准描述,填补领域空白
Prompt-Based Caption Generation for Single-Tooth Dental Images Using Vision-Language Models
- 设计引导式提示框架,提升模型对单牙图像的描述准确性
- 基于RGB图像生成的描述更贴合实际临床场景,质量优于现有方案
- 首次构建面向单颗牙的图文数据生成方法,适合口腔AI研究者使用
数字牙科在深度学习推动下取得显著进展,但现有模型多聚焦于特定任务,如牙齿分割、检测、龋齿识别和牙龈炎分类。缺乏具备整体牙齿知识并能基于该知识进行分析的专用模型。带标注的牙科图像数据集有助于构建此类模型,但现有数据集数量少、范围有限:多数标注描述整个口腔,而图像仅限前牙视图,导致后牙(如磨牙)不清晰,影响标注对视觉语言模型训练的价值。此外,标注通常只关注单一疾病(如牙龈炎),未提供每颗牙的全面评估。由于牙病评分通常针对单颗牙,且正畸中每颗牙独立处理,因此为单颗牙图像生成标注至关重要。据我们所知,尚无此类单颗牙图像与牙科描述配对的数据集。本文旨在通过评估视觉语言模型(VLMs)生成单牙图像描述的可能性与效果,弥补这一空白。结果表明,引导式提示能有效提升生成描述的质量,使提示更准确锚定图像视觉特征。选用RGB图像因其在消费场景中潜力更大。
原文摘要 · Abstract (English)
Digital dentistry has made significant advances with the advent of deep learning. However, the majority of these deep learning-based dental image analysis models focus on very specific tasks such as tooth segmentation, tooth detection, cavity detection, and gingivitis classification. There is a lack of a specialized model that has holistic knowledge of teeth and can perform dental image analysis tasks based on that knowledge. Datasets of dental images with captions can help build such a model. To the best of our knowledge, existing dental image datasets with captions are few in number and limited in scope. In many of these datasets, the captions describe the entire mouth, while the images are limited to the anterior view. As a result, posterior teeth such as molars are not clearly visible, limiting the usefulness of the captions for training vision-language models. Additionally, the captions focus only on a specific disease (gingivitis) and do not provide a holistic assessment of each tooth. Moreover, tooth disease scores are typically assigned to individual teeth, and each tooth is treated as a separate entity in orthodontic procedures. Therefore, it is important to have captions for single-tooth images. As far as we know, no such dataset of single-tooth images with dental captions exists. In this work, we aim to bridge that gap by assessing the possibility of generating captions for dental images using Vision-Language Models (VLMs) and evaluating the extent and quality of those captions. Our findings suggest that guided prompts help VLMs generate meaningful captions. We show that the prompts generated by our framework are better anchored in describing the visual aspects of dental images. We selected RGB images as they have greater potential in consumer scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。