用外部图像记忆引导模型,无需调参就能提升脑成像识别效果
Retrieval-Augmented Visual Prompting: Guiding Foundation Models in Two-Photon Imaging

- 通过检索标注样本作为视觉提示,直接改写输入来引导模型
- 在艾伦脑图谱数据集上,零样本神经元检测准确率显著提升
- 选一个好样本比多个样本更有效,适合快速部署的医学影像场景
双光子钙成像对基础模型构成挑战:图像外观在不同记录和实验条件下差异大,标注数据稀少,且常需快速适应。我们不通过微调调整模型权重,而是探索是否能在推理时通过将外部视觉记忆直接注入输入来引导基础模型。我们基于SAM 3实现该思路,提出检索增强型视觉提示(RAVP)框架:每个目标图像块都附加一个检索到的标注样例,其边界框用作概念提示。RAVP将检索转化为一种视觉提示形式,仅通过输入设计即可实现模型适应。我们研究了多种样例选择策略,包括荧光引导的启发式方法及一个轻量级召回预测器,用于估计哪个样例对目标块最具有信息量。在艾伦脑图谱数据集上的实验表明,样例增强的推理持续提升了零样本神经元检测与实例分割性能。消融实验进一步显示,精心挑选的一个样例比使用多个检索样本更有效。这些结果表明,推理时的视觉记忆注入是基础模型在专业生物医学成像中替代参数调整的一种简单而有效的方案。
原文摘要 · Abstract (English)
Two-photon calcium imaging presents a challenging setting for foundation models: image appearance varies substantially across recordings and experimental conditions, annotations are scarce, and rapid adaptation is often needed. Rather than adapting model weights through fine-tuning, we ask whether a foundation model can be guided at inference time by injecting external visual memory directly into its input. We implement this idea with SAM 3 and introduce Retrieval-Augmented Visual Prompting (RAVP), a framework in which each target tile is augmented with a retrieved annotated exemplar whose bounding box is used as a concept prompt. RAVP turns retrieval into a form of visual prompting and enables adaptation through input design alone. We study multiple exemplar selection strategies, including fluorescence-guided heuristics and a lightweight recall predictor trained to estimate which exemplar is most informative for a target tile. Experiments on the Allen Brain Observatory show that exemplar-augmented inference consistently strengthens zero-shot neuron detection and instance segmentation. Ablation studies further show that a single carefully selected exemplar is more effective than prompting with multiple retrieved examples. These results position inference-time visual memory injection as a simple and effective alternative to parameter adaptation for foundation models in specialized biomedical imaging.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。