基于用户注视点的实例分割，显著降低计算开销。
Foveated Instance Segmentation
- 根据实时注视数据只对关注区域进行分割
- 在ADE20K上达到0.56的IoU，LVIS上0.54
- 适合资源受限的AR/VR设备实时应用
实例分割对增强现实与虚拟现实（AR/VR）至关重要，可实现精准物体识别与交互，提升虚实融合的沉浸感。然而，分割的高计算开销限制了其在资源受限的AR/VR设备上的应用，导致处理延迟大，体验下降。与传统场景不同，AR/VR用户通常仅关注视野中的少数区域，随后转移视角，这为聚焦于注视区域的分割提供了可能。为此，本文提出一种注视区实例分割（FovealSeg）框架，利用实时用户注视数据，仅对关注目标进行分割，大幅降低计算负载并提升实时性能。评估结果表明，FSNet在ADE20K上达到0.56的IoU，在LVIS上达到0.54，显著优于基线模型。代码已开源。
原文摘要 · Abstract (English)
Instance segmentation is essential for augmented reality and virtual reality (AR/VR) as it enables precise object recognition and interaction, enhancing the integration of virtual and real-world elements for an immersive experience. However, the high computational overhead of segmentation limits its application on resource-constrained AR/VR devices, causing large processing latency and degrading user experience. In contrast to conventional scenarios, AR/VR users typically focus on only a few regions within their field of view before shifting perspective, allowing segmentation to be concentrated on gaze-specific areas. This insight drives the need for efficient segmentation methods that prioritize processing instance of interest, reducing computational load and enhancing real-time performance. In this paper, we present a foveated instance segmentation (FovealSeg) framework that leverages real-time user gaze data to perform instance segmentation exclusively on instance of interest, resulting in substantial computational savings. Evaluation results show that FSNet achieves an IoU of 0.56 on ADE20K and 0.54 on LVIS, notably outperforming the baseline. The code is available at https://github.com/SAI-
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。