根据场景动态选择视觉模型子网,实现边缘设备高效零样本推理。
AdaDINO: Context-Adaptive DINO-Distilled Vision Foundation Models for Efficient Open-Vocabulary Edge Inference

- 通过自适应选择最小计算量子网,匹配当前场景任务需求。
- 在相同精度下降低37%平均算力,最高减少74.9%浮点运算。
- 适合资源受限的实时视觉应用,如智能摄像头与可穿戴设备。
始终在线的上下文感知人工智能在边缘设备上运行语言对齐的视觉基础模型(VFMs),其中本地模型是主要的持续计算开销,在严格延迟和功耗限制下尤为突出。由于观察到场景上下文及其相关词汇存在低频变化,本文提出AdaDINO,一种自适应框架,通过匹配当前场景与任务来提升本地VFM推理效率。基于模型规模缩小导致准确率下降依赖任务的现象,将该特性转化为任务级自适应执行。AdaDINO将神经架构搜索(NAS)集成至从DINOv2蒸馏出的语言对齐视觉骨干网络,训练单一子网家族以支持运行时高效执行。云端一个多功能大语言模型(LLM)以低频调用,根据场景上下文精炼候选类别集;同时,一个学习得到的选择器激活预测能保留目标准确率分数的最低成本子网。在保持骨干网络与语义流水线不变的前提下,仅通过学习选择即可在等效分割精度下将平均计算量降低37%。在零样本分类与开放词汇分割任务中,AdaDINO建立了优越的准确率-效率边界:在IN1K上准确率提升最多达7.9%(acc@1),在ADE20K上mIoU提升5.2%,同时在相近精度下平均浮点运算量减少高达74.9%。
原文摘要 · Abstract (English)
Always-on contextual AI runs language-aligned vision foundation models (VFMs) on edge devices, where the on-device model is the dominant continuous compute cost under strict latency and power limits. Due to an observed low-frequency shift in scene context and its relevant vocabulary, we present AdaDINO, an adaptive framework that makes on-device VFM inference efficient by matching execution to the current scene and task. We build on a known phenomenon, that the accuracy drop of shrinking model sizes depends on the task, and turn it into task-level adaptive execution. AdaDINO integrates neural architecture search (NAS) into a language-aligned VFM backbone distilled from DINOv2, training a single family of subnets for efficient execution during runtime. A multimodal large language model (LLM) on the cloud, invoked at low frequency, refines the candidate class set from scene context, while a learned selector activates the least-cost subnet predicted to retain a target fraction of accuracy. With the backbone and semantic pipeline held fixed, learned selection alone reduces average compute by $37\%$ over the best fixed subnet at equal segmentation accuracy. Across zero-shot classification and open-vocabulary segmentation, AdaDINO establishes a strong accuracy-efficiency frontier, improving over evaluated models of comparable sizes by up to $7.9\%$ in acc@1 on IN1K and $5.2\%$ mIoU on ADE20K, and reducing average FLOPs by up to $74.9\%$ at similar accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。