arXiv:2607.24770cs.AIcs.HC2026-07

在边缘设备上实现低延迟、可交互的装配任务指导,兼顾隐私与实时性。

ProcAgent: An Agentic Framework for Procedural Task Guidance on Edge with Human-in-the-Loop

论文配图:ProcAgent: An Agentic Framework for Procedural Task Guidance on Edge with Human-in-the-Loop
图 1 · 摘自论文原文
  • 基于符号任务图与按需视觉语言验证,仅在不确定时调用昂贵推理
  • 端到端运行在Jetson AGX Orin上,文本查询响应约2秒,视觉查询约8秒
  • 支持人机协同确认,用户测试中获高分评价,适合家庭等私密场景

家具组装和家庭维修等程序化任务对用户认知要求高,需理解指令、追踪进度、空间推理并纠错。现有多模态助手多依赖云端推理和持续感知,不适用于注重隐私、低延迟的家庭场景。本文提出ProcAgent,一个完全在单台NVIDIA Jetson AGX Orin设备上运行的、基于视觉的智能体式程序化任务指导系统。其采用提议-验证架构,结合低延迟连续感知、符号化任务图、按需视觉语言验证及基于LLM的交互代理。系统持续推断用户进展,仅在模糊或可能偏离时触发高成本视觉推理,并支持反应式问答与主动干预的人机协同确认。评估涵盖感知准确率、推理能力、任务性能与用户体验四方面。尽管全程本地运行,系统仍保持响应式交互:文本查询约2秒内完成,视觉查询约8秒。10名参与者完成装配任务的用户研究显示,系统在可理解性、可操作性和隐私舒适度上获得正面评价。结果表明,无需牺牲可用性即可在边缘硬件上实现自适应程序化协助。

原文摘要 · Abstract (English)

Procedural tasks such as furniture assembly and home repair impose substantial cognitive demands because users must interpret instructions, track task progress, reason about spatial state, and recover from errors while performing physical actions. Prior multimodal assistants have shown promise for procedural guidance, but most rely on cloud inference and fixed always-on perception, making them poorly suited to privacy-sensitive, latency-critical domestic settings. We present ProcAgent, a fully on-device, agentic, vision-based procedural assistant for real-time adaptive guidances on a single NVIDIA Jetson AGX Orin. ProcAgent uses a propose-and-verify architecture that combines low-latency continuous perception, a symbolic task graph, on-demand vision-language verification, and an LLM-based interaction agent. The system continuously proposes user progress, invokes expensive visual reasoning only when ambiguity or likely deviation arises, and supports both reactive question answering and proactive intervention with human-in-the- loop confirmation. We evaluate ProcAgent along four dimensions: perception accuracy, reasoning, task-level performance, and user experience. Despite running entirely on-device, the system maintains responsive interaction, resolving text-only queries in approximately 2 seconds and visually grounded queries in approximately 8 seconds. In a user study with 10 participants completing assembly tasks, ProcAgent receives positive ratings for comprehensibility, actionability, and privacy comfort. These results show that adaptive procedural assistance can be achieved entirely on edge hardware without sacrificing usability.

边缘计算任务指导人机协同视觉语言

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。