arXiv:2410.11722cs.CVcs.AI2024-10NeurIPS被引 5

构建真实点击数据集,评估交互式分割模型在真实用户行为下的表现。

RClicks: Realistic Click Simulation for Benchmarking Interactive Segmentation

  • 基于47.5万真实用户点击数据,构建可模拟真实行为的点击模型。
  • 发现多数模型在真实点击下性能下降,且对点击模式不鲁棒。
  • 适合关注真实场景部署效果的研究者和开发者。

Segment Anything (SAM) 的出现引发了对交互式分割的研究热潮,尤其在图像编辑和加速数据标注方面。与常规语义分割不同,交互式分割允许用户通过提示(如点击)直接影响输出结果。然而,真实场景中的点击模式尚未被充分研究。现有方法多假设用户会点击最大错误区域的中心,但最新研究表明这并不总是成立。因此,尽管基线基准测试中表现优异,实际部署时模型可能表现不佳。为更真实地模拟用户点击,我们开展大规模众包研究,收集了47.5万条真实用户点击数据。借鉴显著性任务思想,提出点击可得性模型,实现贴近真实用户输入的点击采样。基于该模型与数据集,我们构建了RClicks基准,用于全面评估现有交互式分割方法在真实点击下的表现。不仅评估平均性能,还考察对点击模式的鲁棒性。结果显示,真实使用中模型表现可能远低于基线报告,多数方法缺乏鲁棒性。我们认为,RClicks是迈向真实场景最优用户体验的重要一步。

原文摘要 · Abstract (English)

The emergence of Segment Anything (SAM) sparked research interest in the field of interactive segmentation, especially in the context of image editing tasks and speeding up data annotation. Unlike common semantic segmentation, interactive segmentation methods allow users to directly influence their output through prompts (e.g. clicks). However, click patterns in real-world interactive segmentation scenarios remain largely unexplored. Most methods rely on the assumption that users would click in the center of the largest erroneous area. Nevertheless, recent studies show that this is not always the case. Thus, methods may have poor performance in real-world deployment despite high metrics in a baseline benchmark. To accurately simulate real-user clicks, we conducted a large crowdsourcing study of click patterns in an interactive segmentation scenario and collected 475K real-user clicks. Drawing on ideas from saliency tasks, we develop a clickability model that enables sampling clicks, which closely resemble actual user inputs. Using our model and dataset, we propose RClicks benchmark for a comprehensive comparison of existing interactive segmentation methods on realistic clicks. Specifically, we evaluate not only the average quality of methods, but also the robustness w.r.t. click patterns. According to our benchmark, in real-world usage interactive segmentation models may perform worse than it has been reported in the baseline benchmark, and most of the methods are not robust. We believe that RClicks is a significant step towards creating interactive segmentation methods that provide the best user experience in real-world cases.

交互式分割点击模拟真实场景基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。