针对AR眼镜相机不稳问题，提出无需预处理的高效深度估计算法。
Efficient Depth Estimation for Unstable Stereo Camera Systems on AR Glasses
- 用位置编码的单应性矩阵网络替代传统校正，直接处理未对齐图像。
- 以分组点卷积和归一化点积近似替代代价体，降低计算开销。
- 模型端到端延迟减少44.5%，适合部署在带GPU/NPU的AR设备上。
立体深度估计是增强现实（AR）的核心组件，需满足低延迟实时处理要求。然而，传统预处理如校正和非机器学习计算（如代价体）的延迟远超深度模型本身，阻碍了实时性。为此，我们设计了适配现代硬件（GPU与NPU）的新型方法：通过引入带有校正位置编码（RPE）的单应性矩阵预测网络，消除预处理步骤，实现对未校正图像的低延迟、鲁棒处理；用基于分组点卷积的算子及基于层归一化与点积的余弦相似度近似替代代价体。在此基础上，构建了多头深度模型（MultiHeadDepth）与无预处理版本（HomoDepth）。相较于工业界先进模型，MultiHeadDepth在准确率上提升11.8%-30.3%，延迟降低22.9%-25.2%；HomoDepth可直接处理未校正图像，端到端延迟减少44.5%。此外，结合多任务学习处理输入错位问题，使绝对相对误差降低10.0%-24.3%。实验验证了方法的有效性，在降低延迟的同时提升了性能。代码已开源。
原文摘要 · Abstract (English)
Stereo depth estimation is a fundamental component in augmented reality (AR), which requires low latency for real-time processing. However, preprocessing such as rectification and non-ML computations such as cost volume require significant amount of latency exceeding that of an ML model itself, which hinders the real-time processing required by AR. Therefore, we develop alternative approaches to the rectification and cost volume that consider ML acceleration (GPU and NPUs) in recent hardware. For pre-processing, we eliminate it by introducing homography matrix prediction network with a rectification positional encoding (RPE), which delivers both low latency and robustness to unrectified images. For cost volume, we replace it with a group-pointwise convolution-based operator and approximation of cosine similarity based on layernorm and dot product. Based on our approaches, we develop MultiHeadDepth (replacing cost volume) and HomoDepth (MultiHeadDepth + removing pre-processing) models. MultiHeadDepth provides 11.8-30.3% improvements in accuracy and 22.9-25.2% reduction in latency compared to a state-of-the-art depth estimation model for AR glasses from industry. HomoDepth, which can directly process unrectified images, reduces the end-to-end latency by 44.5%. We also introduce a multi-task learning method to handle misaligned stereo inputs on HomoDepth, which reduces the AbsRel error by 10.0-24.3%. The overall results demonstrate the efficacy of our approaches, which not only reduce the inference latency but also improve the model performance. Our code is available at https://github.com/UCI-ISA-Lab/MultiHeadDepth-HomoDepth
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。