用类脑芯片实现毫秒级光流计算,速度超人类400倍。
Neuromorphic spatiotemporal optical flow: Enabling ultrafast visual perception beyond human capabilities
- 将时间信息嵌入二维晶体管阵列,实现时空联合分析
- 1-2毫秒内完成运动区域定位,推理速度提升400%
- 适合高速机器人、自动驾驶等对实时性要求极高的场景
光流算法受生物视觉启发,可计算视觉场景中的运动矢量,使机器人在复杂动态环境中表现优异。然而现有算法在实际部署中存在约0.6秒/次的延迟(为人类处理速度的4倍),难以满足实时需求。本文提出一种类脑时空光流方法,通过二维范德华异质结构中的浮栅突触晶体管直接编码时间信息,辅助空间运动分析。相比传统仅依赖空间信息的方法,该系统利用嵌入式时间线索,可在1-2毫秒内快速识别运动兴趣区域,实现视觉输入的动态筛选,从而加速速度计算与任务执行。硬件层面,由于各功能层间原子级平整界面,突触晶体管具备约100微秒的高频响应、超过10000秒的非挥发性及超过8000次循环的优异耐久性,支持稳定视觉处理。软件基准测试显示,系统相较当前最优算法实现400%的速度提升,频繁超越人类水平性能,同时保持或提升精度,充分利用了嵌入式时间先验信息。
原文摘要 · Abstract (English)
Optical flow, inspired by the mechanisms of biological visual systems, calculates spatial motion vectors within visual scenes that are necessary for enabling robotics to excel in complex and dynamic working environments. However, current optical flow algorithms, despite human-competitive task performance on benchmark datasets, remain constrained by unacceptable time delays (~0.6 seconds per inference, 4X human processing speed) in practical deployment. Here, we introduce a neuromorphic optical flow approach that addresses delay bottlenecks by encoding temporal information directly in a synaptic transistor array to assist spatial motion analysis. Compared to conventional spatial-only optical flow methods, our spatiotemporal neuromorphic optical flow offers the spatial-temporal consistency of motion information, rapidly identifying regions of interest in as little as 1-2 ms using the temporal motion cues derived from the embedded temporal information in the two-dimensional floating gate synaptic transistors. Thus, the visual input can be selectively filtered to achieve faster velocity calculations and various task execution. At the hardware level, due to the atomically sharp interfaces between distinct functional layers in two-dimensional van der Waals heterostructures, the synaptic transistor offers high-frequency response (~100 μs), robust non-volatility (>10000 s), and excellent endurance (>8000 cycles), enabling robust visual processing. In software benchmarks, our system outperforms state-of-the-art algorithms with a 400% speedup, frequently surpassing human-level performance while maintaining or enhancing accuracy by utilizing the temporal priors provided by the embedded temporal information.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。