用轻量缝合层连接不同视觉大模型,实现高效集成与性能提升。
Revisiting Model Stitching In the Foundation Model Era
- 在目标模型倒数第二层使用特征匹配损失,显著提升异构模型缝合成功率。
- 深层缝合点下,组合模型性能优于任一原始模型,推理开销仅小幅增加。
- 提出VFM缝合树架构,支持多视觉模型共享早期特征,灵活权衡精度与延迟。
模型缝合通过轻量缝合层将一个模型(源)的前几层与另一个模型(目标)的后几层连接,用于探测表征兼容性。以往研究发现,即使初始化或目标函数不同,同一数据集训练的模型仍可无缝缝合(准确率下降可忽略)。我们重新审视了在视觉基础模型(VFMs)时代下的缝合问题,这些模型在目标、数据和模态混合上存在差异(如CLIP、DINOv2、SigLIP 2),并提出系统性协议,涵盖缝合点、缝合层类型、训练损失及下游任务。三个核心发现:(1) 缝合层训练至关重要:传统方法在缝合点匹配中间特征或端到端优化任务损失,在浅层缝合时难以保持准确率;(2) 在目标模型倒数第二层引入简单特征匹配损失,可使异构视觉基础模型在多种视觉任务中稳定缝合;(3) 深层缝合时,缝合模型性能可超越任一原始模型,且仅带来微小推理开销(仅缝合层)。基于此,我们提出视觉基础模型缝合树(VFM Stitch Tree, VST),共享多个视觉模型的早期层,保留各自后期层,为常需调用多个视觉模型的多模态大模型提供可控的精度-延迟权衡。本研究将缝合从诊断工具升级为整合互补视觉模型优势的实际方法,并精准定位其表征对齐与分歧点。
原文摘要 · Abstract (English)
Model stitching, connecting early layers of one model (source) to later layers of another (target) via a light stitch layer, has served as a probe of representational compatibility. Prior work finds that models trained on the same dataset remain stitchable (negligible accuracy drop) despite different initializations or objectives. We revisit stitching for Vision Foundation Models (VFMs) that vary in objectives, data, and modality mix (e.g., CLIP, DINOv2, SigLIP 2) and ask: Are heterogeneous VFMs stitchable? We introduce a systematic protocol spanning the stitch points, stitch layer families, training losses, and downstream tasks. Three findings emerge. (1) Stitch layer training matters: conventional approaches that match the intermediate features at the stitch point or optimize the task loss end-to-end struggle to retain accuracy, especially at shallow stitch points. (2) With a simple feature-matching loss at the target model's penultimate layer, heterogeneous VFMs become reliably stitchable across vision tasks. (3) For deep stitch points, the stitched model can surpass either constituent model at only a small inference overhead (for the stitch layer). Building on these findings, we further propose the VFM Stitch Tree (VST), which shares early layers across VFMs while retaining their later layers, yielding a controllable accuracy-latency trade-off for multimodal LLMs that often leverage multiple VFMs. Taken together, our study elevates stitching from a diagnostic probe to a practical recipe for integrating complementary VFM strengths and pinpointing where their representations align or diverge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。