arXiv:2412.03118cs.HCcs.CV2024-12被引 4

盲人可用自然语言搜索物体,系统实时提供定位与互动信息。

ObjectFinder: An Open-Vocabulary Assistive System for Interactive Object Search by Blind People

  • 用开放词汇模型支持自然语言查询物体
  • 实时提供距离、方向及多场景信息获取能力
  • 与视障用户共创,提升交互自主性

视障人士在陌生环境中寻找物体是一项挑战性任务,涉及目标指定、检测和根据用户意图获取详细信息。现有基于描述和检测的辅助技术未能充分支持交互式搜索的多维度需求。我们提出ObjectFinder,一个面向视障人士的开放词汇可穿戴交互式物体搜索系统。该系统允许用户使用灵活表述查询目标物体,一旦检测到目标,即可实时提供第一视角的定位信息(包括距离和方向)。用户可根据意图发起不同分支,如导航至目标或感知周围环境。ObjectFinder结合了开放词汇目标检测器与多模态大语言模型,设计过程与一位视障合作者共同完成。通过八名视障用户的探索性研究,对比BeMyAI和Google Lookout两款主流辅助应用,结果显示多数参与者认为ObjectFinder提升了独立性,更偏好其在场景理解、导航支持和主动识别方面的表现。最后讨论了未来辅助系统在交互式物体搜索中的设计启示。

原文摘要 · Abstract (English)

Searching for objects in unfamiliar scenarios is a challenging task for blind people. It involves specifying the target object, detecting it, and then gathering detailed information according to the user's intent. However, existing description- and detection-based assistive technologies do not sufficiently support the multifaceted nature of interactive object search tasks. We present ObjectFinder, an open-vocabulary wearable assistive system for interactive object search by blind people. ObjectFinder allows users to query target objects using flexible wording. Once the target object is detected, it provides egocentric localization information in real-time, including distance and direction. Users can then initiate different branches to gather detailed information based on their intent towards the target object, such as navigating to it or perceiving its surroundings. ObjectFinder is powered by a seamless combination of open-vocabulary models, namely an open-vocabulary object detector and a multimodal large language model. The ObjectFinder design concept and its development were carried out in collaboration with a blind co-designer. To evaluate ObjectFinder, we conducted an exploratory user study with eight blind participants. We compared ObjectFinder to BeMyAI and Google Lookout, popular description- and detection-based assistive applications. Our findings indicate that most participants felt more independent with ObjectFinder and preferred it for object search, as it enhanced scene context gathering and navigation, and allowed for active target identification. Finally, we discuss the implications for future assistive systems to support interactive object search.

辅助系统开放词汇视障技术

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。