去掉光流估计中的代价体，提速6倍且内存降为1/6
Removing Cost Volumes from Optical Flow Estimators
- 训练时逐步移除代价体，让网络自适应不依赖它
- 最快模型达20帧/秒，仅需500MB显存处理全高清视频
- 适合实时应用或资源受限场景的光流系统
现代光流估计器普遍使用代价体，但其计算与存储开销限制了处理速度和输入分辨率。我们观察到，当基于RAFT的网络结构充分训练后，代价体的重要性显著下降。为此提出一种训练策略，可在整个训练过程中移除代价体。由此构建的三个模型覆盖不同算力需求：最精确模型达到当前最佳性能，推理速度提升1.2倍，内存占用降低6倍；最快模型在仅500MB GPU内存下实现全高清帧20帧/秒的处理速度。
原文摘要 · Abstract (English)
Cost volumes are used in every modern optical flow estimator, but due to their computational and space complexity, they are often a limiting factor regarding both processing speed and the resolution of input frames. Motivated by our empirical observation that cost volumes lose their importance once all other network parts of, e.g., a RAFT-based pipeline have been sufficiently trained, we introduce a training strategy that allows removing the cost volume from optical flow estimators throughout training. This leads to significantly improved inference speed and reduced memory requirements. Using our training strategy, we create three different models covering different compute budgets. Our most accurate model reaches state-of-the-art accuracy while being $1.2\times$ faster and having a $6\times$ lower memory footprint than comparable models; our fastest model is capable of processing Full HD frames at $20\,\mathrm{FPS}$ using only $500\,\mathrm{MB}$ of GPU memory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。