通过对象中心特征与层级提示,提升视觉语言模型对复杂手术动作的零样本识别能力。
fine-CLIP: Enhancing Zero-Shot Fine-Grained Surgical Action Recognition with Vision-Language Models
- 采用层级提示与图聚类,捕捉手术动作的细粒度语义结构
- 在CholecT50数据集上,新设定下F1和mAP显著提升
- 适合需要零样本泛化能力的医疗视觉研究者
尽管像CLIP这样的视觉语言模型在零样本手术阶段识别上取得进展,但在细粒度手术动作(尤其是动作三元组)识别上仍表现不佳。原因在于现有CLIP方法依赖全局图像特征,忽视了复杂任务中关键的细粒度语义与上下文细节,且未利用三元组内在的层次结构,限制了对新三元组的泛化能力。为此,我们提出fine-CLIP,通过学习对象中心特征并利用三元组的层次结构来改进。方法包含三个组件:层次化提示建模以捕捉共享语义,基于LoRA的视觉主干适配以增强特征提取,以及基于图的特征压缩策略,将相似的图像块特征聚类为有意义的对象簇。针对三元组分类的挑战,我们在CholecT50数据集上引入两种新的基线到新类的泛化评估设置:Unseen-Target(评估对新解剖结构三元组的适应性)和Unseen-Instrument-Verb(需泛化至新器械-动词组合)。实验表明,fine-CLIP在F1和mAP指标上均有显著提升,有效增强了对新型手术三元组的零样本识别能力。
原文摘要 · Abstract (English)
While vision-language models like CLIP have advanced zero-shot surgical phase recognition, they struggle with fine-grained surgical activities, especially action triplets. This limitation arises because current CLIP formulations rely on global image features, which overlook the fine-grained semantics and contextual details crucial for complex tasks like zero-shot triplet recognition. Furthermore, these models do not explore the hierarchical structure inherent in triplets, reducing their ability to generalize to novel triplets. To address these challenges, we propose fine-CLIP, which learns object-centric features and leverages the hierarchy in triplet formulation. Our approach integrates three components: hierarchical prompt modeling to capture shared semantics, LoRA-based vision backbone adaptation for enhanced feature extraction, and a graph-based condensation strategy that groups similar patch features into meaningful object clusters. Since triplet classification is a challenging task, we introduce an alternative yet meaningful base-to-novel generalization benchmark with two settings on the CholecT50 dataset: Unseen-Target, assessing adaptability to triplets with novel anatomical structures, and Unseen-Instrument-Verb, where models need to generalize to novel instrument-verb interactions. fine-CLIP shows significant improvements in F1 and mAP, enhancing zero-shot recognition of novel surgical triplets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。