构建混合现实协作数据集，推动人机协同研究
SigmaCollab: An Application-Driven Dataset for Physically Situated Collaboration
- 采集85个会话的多模态数据，含追踪与摄像头信息
- 约14小时数据，覆盖无训练用户在真实任务中的交互
- 适合研究人机协作、混合现实辅助系统的技术团队
我们提出SigmaCollab，一个支持物理情境下人机协作研究的数据集。该数据集包含85个会话，未受训参与者在混合现实辅助AI代理引导下完成现实世界操作任务。数据涵盖丰富的多模态流：参与者与系统音频、头戴设备的视点摄像头画面、深度图、头部、手部及注视追踪信息，以及事后标注。尽管数据量较小（约14小时），但其应用驱动和交互特性凸显了人机协作的新挑战，并为各类运行于该场景的AI模型提供了更真实的测试环境。未来工作将基于此构建混合现实任务辅助场景下的协作基准。SigmaCollab可在https://github.com/microsoft/SigmaCollab获取。
原文摘要 · Abstract (English)
We introduce SigmaCollab, a dataset enabling research on physically situated human-AI collaboration. The dataset consists of a set of 85 sessions in which untrained participants were guided by a mixed-reality assistive AI agent in performing procedural tasks in the physical world. SigmaCollab includes a set of rich, multimodal data streams, such as the participant and system audio, egocentric camera views from the head-mounted device, depth maps, head, hand and gaze tracking information, as well as additional annotations performed post-hoc. While the dataset is relatively small in size (~ 14 hours), its application-driven and interactive nature brings to the fore novel research challenges for human-AI collaboration, and provides more realistic testing grounds for various AI models operating in this space. In future work, we plan to use the dataset to construct a set of benchmarks for physically situated collaboration in mixed-reality task assistive scenarios. SigmaCollab is available at https://github.com/microsoft/SigmaCollab.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。