用视频解码器的运动矢量预对齐，实现手机端毫秒级视频插帧。
ANVIL: Accelerator-Native Video Interpolation via Codec Motion Vector Priors
- 复用H.264解码器的运动矢量预对齐帧，省去学习光流和采样模块
- 1080p下8比特整数推理仅需12.8毫秒，满足实时性要求
- 适合移动端部署，尤其适配已有解码器支持的H.264视频
在移动端神经处理单元（NPUs）上实现30到60帧每秒的实时视频插帧，要求每帧合成时间不超过33.3毫秒。我们发现主流基于光流的插帧方法在移动NPUs上面临三大结构部署障碍：空间采样操作超出帧预算或缺乏硬件支持，迭代光流优化在8比特整数量化后崩溃，且内存密集型操作主导推理图。ANVIL通过复用H.264/AVC解码器中的运动矢量进行输入帧预对齐，移除学习光流、空间采样与迭代累积模块，使剩余残差由以卷积为主的计算密集型网络处理。在高通骁龙8 Gen 3设备上，ANVIL实现1080p下12.8毫秒的8比特整数精度推理；开源Android播放器在30分钟连续播放中保持28.4毫秒的中位端到端延迟。每算子因果分析揭示，递归光流状态的量化累加是迭代方法在整数量化下失败的关键机制。当前设计针对具有解码器暴露运动矢量的H.264/AVC播放场景。
原文摘要 · Abstract (English)
Real-time 30-to-60 fps video frame interpolation on mobile neural processing units (NPUs) requires each synthesized frame within 33.3 ms. We show that mainstream flow-based video frame interpolation faces three structural deployment barriers on mobile NPUs: spatial sampling operators exceed the frame budget or lack hardware support, iterative flow refinement collapses under 8-bit integer post-training quantization, and memory-bound operators dominate the inference graph. ANVIL addresses these barriers by reusing motion vectors from the H.264/AVC decoder to prealign input frames, removing learned optical flow, spatial sampling, and iterative accumulation from the accelerator graph. The remaining residual is refined by a convolution-dominated network composed almost entirely of compute-bound operators. On a Snapdragon 8 Gen 3 device, ANVIL achieves 12.8 ms 1080p inference at 8-bit integer precision; an open-source Android player sustains 28.4 ms median end-to-end latency over 30-minute continuous playback. Per-operator causal analysis identifies quantized accumulation on recurrent flow states as a key mechanism behind integer quantization failure in iterative methods. The current design targets H.264/AVC playback with decoder-exposed motion vectors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。