用轻量方法让通用视觉模型精准识别肠镜息肉,无需修改原模型。
From Generalist to Specialist: A Context-Fusion Framework for Endoscopic Polyp Reporting with a Frozen VLM

- 通过隐式指令与显式证据融合,不改原模型参数实现专业化。
- 在2056张图像上表现最佳,错误率比基线低70.5%。
- 仅增加0.006%参数,适合医疗领域快速部署。
可靠的内镜息肉报告需要整合定量病变大小、标准化巴黎分类和临床有意义的形态描述。通用视觉语言模型(VLM)为图像理解与报告生成提供统一接口。然而,现有专业化策略通常依赖任务专用模型或权重调整,未能解决如何在保留统一接口和预训练能力的同时引入可靠专家知识。本文提出一种上下文融合框架,通过隐式指令上下文和显式转换上下文对冻结的通用VLM进行专业化,无需修改其预训练权重。具体地,自监督息肉编码器检索相关图文对作为特定查询的显式证据,而学习到的连续专家标记提供跨病例共享的隐式指令上下文。在2056张专家标注的公开内镜图像上进行实验,对比了通用VLM、任务专用预测器和权重调整方法,评估了专业化性能、统一报告能力和适应效率。在数值、类别和报告生成指标上,该框架显著优于直接冻结VLM推理,并在所有方法中取得最优综合表现。仅增加相当于冻结VLM参数量0.006%的可训练参数。当检索到的前1个案例包含正确目标类别时,本框架纠正了70.5%的权重调整基线错误。这些结果表明,上下文融合框架是冻结VLM专业化的一种轻量高效策略。
原文摘要 · Abstract (English)
Reliable endoscopic polyp reporting requires integrating quantitative lesion sizing, standardized Paris classification, and clinically meaningful morphological description within a single record. General-purpose vision-language models (VLMs) offer a unified interface for image understanding and report generation. Existing specialization strategies, however, typically rely on task-specific models or model-weight adaptation, leaving unresolved how to introduce reliable specialist knowledge while preserving both this unified interface and the VLM's pretrained capabilities. We introduce a context-fusion framework that specializes a frozen general-purpose VLM through both implicit instruction context and explicit transduction context without modifying its pretrained weights. Specifically, a self-supervised polyp encoder retrieves related image-report pairs as explicit, query-specific evidence, while learned continuous specialist tokens provide implicit instruction context shared across cases. Experiments were conducted on 2,056 expert-annotated public endoscopic images. We compared the framework with general-purpose VLMs, task-specific predictors, and weight-adaptation methods to assess specialist performance, unified reporting, and adaptation efficiency. Across numerical, categorical, and report-generation metrics, the proposed framework substantially improved direct frozen-VLM inference and achieved the strongest overall performance among the evaluated methods. It added trainable parameters equal to only 0.006% of the frozen VLM's parameter count. When the top-1 retrieved case carried the correct target category, our framework corrected 70.5% of the errors made by a weight-adaptation baseline. These findings support the context-fusion framework as a lightweight and effective strategy for specialist adaptation of a frozen VLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。