用对数极坐标机制实现全高清视频快速识别,模拟人眼聚焦
HiResNets: Native Full-HD Video Recognition with Foveal Residual Streams

- 通过对数极坐标变换在残差流中保持高分辨率,避免内存爆炸
- 在小物体和细粒度识别任务上显著优于传统方法,准确率提升明显
- 适合需要高分辨率处理的头戴式视频分析场景
近年来图像与视频识别的进步往往以高昂的内存开销为代价:模型变大、分辨率提高、时间上下文延长。其根本原因在于卷积网络与视觉变压器采用网格采样,导致内存与计算量随分辨率呈二次或更高增长。本文提出HiResNets,一种残差块具有对数平方增长特性的网络,可快速处理极高分辨率视频。核心思想是将残差流作为高分辨率缓冲区,卷积层仅通过对数极坐标映射读写该流。各层自适应聚焦于帧中不同区域,仅在关注点附近保持极高分辨率。整个高分辨率表征在残差流中逐步构建,类似生物视觉中的眼动扫描形成完整画面。文中还提出了理论构造,彻底消除残差流分辨率的二次依赖。实验表明,HiResNets能学习像人类视觉一样聚焦,在困难的自指视角视频识别任务中表现优异,尤其在包含小物体和细粒度识别的任务中优势明显。
原文摘要 · Abstract (English)
Much of the recent progress in image and video recognition has come at the cost of memory: larger models, increased resolution, and longer temporal contexts. An inevitable component is the quadratic (or larger) growth of memory and compute based on image resolution, which is a property of the grid sampling used in convolutional networks and vision transformers. In this work we study residual networks whose convolutional blocks have logarithmic-square growth instead, enabling them to process very high-resolution video quickly. The key insight is to use a residual architecture's residual stream as a high-resolution buffer, to which convolutional blocks only read and write via log-polar image warp operations. Layers adaptively focus on different parts of each frame, with very high resolution only near the focus point. A complete high-resolution representation is built up in the residual stream, analogous to eye saccades creating a complete picture in biological vision, and a theoretical construction is presented that eliminates the quadratic dependency of the residual stream resolution. Experiments demonstrate that our proposed HiResNets learn to foveate around scenes similarly to human vision, and have superior performance in difficult egocentric video recognition tasks, especially egocentric video with small objects and fine-grained recognition.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。