用iPhone 15 Pro Max的激光雷达实现高效隐私保护的背景虚化
Efficient and Privacy-Protecting Background Removal for 2D Video Streaming using iPhone 15 Pro Max LiDAR
- 利用手机激光雷达获取深度信息，不依赖光照条件
- 实现实时60帧每秒的背景移除，深度图分辨率320x240
- 适合视频会议、直播等注重隐私与实时性的场景
消费级移动设备中的激光雷达（LiDAR）技术可替代传统背景移除与合成方法。与色键抠像及训练好的AI模型不同，激光雷达的深度信息不受主体光照影响，在低光和强光环境下表现一致。我们结合iPhone 15 Pro Max的激光雷达与彩色摄像头，通过GPU进行图像处理。使用Apple的SwiftUI和Swift框架开发用户界面与后端，以Metal Shader Language（MSL）实现在标准60帧每秒流媒体速率下的实时图像增强。该技术的主要限制是深度数据的传输带宽，导致深度图分辨率降至320x240；此外，部分材质对激光雷达红外激光反射不准确。若未来移动设备激光雷达分辨率能与彩色图像一致，其有望成为视频应用与摄影中主流的背景移除方案。
原文摘要 · Abstract (English)
Light Detection and Ranging (LiDAR) technology in consumer-grade mobile devices can be used as a replacement for traditional background removal and compositing techniques. Unlike approaches such as chroma keying and trained AI models, LiDAR's depth information is independent of subject lighting, and performs equally well in low-light and well-lit environments. We integrate the LiDAR and color cameras on the iPhone 15 Pro Max with GPU-based image processing. We use Apple's SwiftUI and Swift frameworks for user interface and backend development, and Metal Shader Language (MSL) for realtime image enhancement at the standard iPhone streaming frame rate of 60 frames per second. The only meaningful limitations of the technology are the streaming bandwidth of the depth data, which currently reduces the depth map resolution to 320x240, and any pre-existing limitations of the LiDAR IR laser to reflect accurate depth from some materials. If the LiDAR resolution on a mobile device like the iPhone can be improved to match the color image resolution, LiDAR could feasibly become the preeminent method of background removal for video applications and photography.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。