arXiv:2601.21362cs.NIcs.CV2026-01

针对移动端视频分析中ViT模型推理慢的问题,提出动态混合分辨率策略提升速度与精度平衡。

ViTMAlis: Towards Latency-Critical Mobile Video Analytics with Vision Transformers

  • 采用动态混合分辨率推理,根据内容和网络条件灵活调整输入分辨率
  • 在密集预测任务中将端到端延迟降低40%以上,同时提升渲染准确率
  • 专为ViT设计的设备-边缘协同框架,适合实时视频分析场景

边缘辅助的移动视频分析(MVA)应用正从基于卷积神经网络(CNN)的模型转向视觉变换器(ViTs),以利用其更强的全局上下文建模和泛化能力。然而,在延迟敏感的MVA场景中部署这些先进模型面临重大挑战。与传统以网络传输为主瓶颈的CNN卸载范式不同,基于ViT的系统受限于显著的推理延迟,尤其在需要高分辨率输入的密集预测任务中,其固有的二次方计算复杂度被进一步放大。为此,我们提出一种面向ViT骨干网络密集预测模型的动态混合分辨率推理策略,支持运行时灵活权衡速度与精度。在此基础上,我们设计了ViTMAlis——一个原生支持ViT的设备-边缘卸载框架,可动态适应网络状态与视频内容,联合减少传输与推理延迟。我们在商用移动与边缘设备上实现了完整的ViTMAlis原型系统。大量实验表明,相比最先进的以精度为中心、内容感知及延迟自适应基线方法,ViTMAlis显著降低了端到端卸载延迟,同时提升了用户感知的渲染准确率,为下一代移动智能提供了可行基础。

原文摘要 · Abstract (English)

Edge-assisted mobile video analytics (MVA) applications are increasingly shifting from using vision models based on convolutional neural networks (CNNs) to those built on vision transformers (ViTs) to leverage their superior global context modeling and generalization capabilities. However, deploying these advanced models in latency-critical MVA scenarios presents significant challenges. Unlike traditional CNN-based offloading paradigms where network transmission is the primary bottleneck, ViT-based systems are constrained by substantial inference delays, particularly for dense prediction tasks where the need for high-resolution inputs exacerbates the inherent quadratic computational complexity of ViTs. To address these challenges, we propose a dynamic mixed-resolution inference strategy tailored for ViT-backboned dense prediction models, enabling flexible runtime trade-offs between speed and accuracy. Building on this, we introduce ViTMAlis, a ViT-native device-to-edge offloading framework that dynamically adapts to network conditions and video content to jointly reduce transmission and inference delays. We implement a fully functional prototype of ViTMAlis on commodity mobile and edge devices. Extensive experiments demonstrate that, compared to state-of-the-art accuracy-centric, content-aware, and latency-adaptive baselines, ViTMAlis significantly reduces end-to-end offloading latency while improving user-perceived rendering accuracy, providing a practical foundation for next-generation mobile intelligence.

视频分析视觉变换器边缘计算低延迟

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。