arXiv:2507.21072cs.HCcs.RO2025-07被引 5

为工厂打造离线运行的智能助手,支持实时识别与语音交互。

Snap, Segment, Deploy: A Visual Data and Detection Pipeline for Wearable Industrial Assistants

论文配图:Snap, Segment, Deploy: A Visual Data and Detection Pipeline for Wearable Industrial Assistants
图 1 · 摘自论文原文
  • 在设备端集成检测、语音与生成模型,实现无云依赖的辅助
  • 在Gear8数据集上提升对域偏移和视觉退化的鲁棒性
  • 适合需要隐私保护与低延迟的工业现场部署

工业装配任务日益复杂,需快速适应多样流程与组件,但工厂环境常受限于计算能力、网络连接及严格隐私要求,传统云端或全自主方案难以落地。本文提出一种基于移动端的工业辅助系统,通过设备端感知与语音接口实现实时、半免手持交互。系统将轻量级目标检测、语音识别与检索增强生成(RAG)整合为模块化本地处理流水线,全程离线运行,无需人工标注或云服务即可提供零件操作与流程理解支持。为实现可扩展训练,采用自动化数据构建流程,并引入两阶段精炼策略,提升在域偏移下的视觉鲁棒性。在自建数据集Gear8上的实验表明,系统显著增强对域偏移及常见视觉退化的容忍度。结构化用户研究进一步验证其实际可行性,用户对指引清晰度与交互质量反馈积极。结果表明,该框架为工业场景提供了可部署的实时、隐私保护智能辅助方案。论文接受后将开源Gear8数据集与源代码。

原文摘要 · Abstract (English)

Industrial assembly tasks increasingly demand rapid adaptation to complex procedures and varied components, yet are often conducted in environments with limited computing, connectivity, and strict privacy requirements. These constraints make conventional cloud-based or fully autonomous solutions impractical for factory deployment. This paper introduces a mobile-device-based assistant system for industrial training and operational support, enabling real-time, semi-hands-free interaction through on-device perception and voice interfaces. The system integrates lightweight object detection, speech recognition, and Retrieval-Augmented Generation (RAG) into a modular on-device pipeline that operates entirely on-device, enabling intuitive support for part handling and procedure understanding without relying on manual supervision or cloud services. To enable scalable training, we adopt an automated data construction pipeline and introduce a two-stage refinement strategy to improve visual robustness under domain shift. Experiments on our generated dataset, i.e., Gear8, demonstrate improved robustness to domain shift and common visual corruptions. A structured user study further confirms its practical viability, with positive user feedback on the clarity of the guidance and the quality of the interaction. These results indicate that our framework offers a deployable solution for real-time, privacy-preserving smart assistance in industrial environments. We will release the Gear8 dataset and source code upon acceptance.

工业辅助离线推理RAG语音交互

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。