高精度标注的厨房行为数据集，助力理解真实场景下的人类动作与认知。
EPFL-Smart-Kitchen-30: Densely annotated cooking dataset with 3D kinematics to challenge video and language models
- 多模态同步采集手、体、眼动3D轨迹，覆盖29.7小时烹饪视频
- 每分钟33.78个动作片段，密集标注支持细粒度分析
- 提供视觉-语言、动作生成等4项基准任务，适合行为建模研究者
理解人类行为需要能捕捉复杂任务中人的真实表现的数据集。厨房是评估人类运动与认知功能的理想环境，因其中自然包含切菜、清洁等多种复杂动作。本文介绍EPFL-Smart-Kitchen-30数据集，基于非侵入式动作捕捉平台在厨房环境中采集。使用九台静态RGB-D相机、惯性测量单元（IMUs）及一台头戴式HoloLens~2设备，同步记录3D手部、身体与眼球运动。该数据集为多视角动作数据，涵盖29.7小时、16名受试者烹饪四种食谱，包含外视角、内视角、深度图、IMUs、眼动、身体与手部运动学信息。动作序列以每分钟33.78个动作段进行密集标注。基于此多模态数据，我们提出四项基准：1）视觉-语言理解；2）语义文本到动作生成；3）多模态动作识别；4）基于姿态的动作分割。期望该数据集推动更优方法的发展，并深化对生态有效人类行为本质的理解。代码与数据见https://github.com/amathislab/EPFL-Smart-Kitchen
原文摘要 · Abstract (English)
Understanding behavior requires datasets that capture humans while carrying out complex tasks. The kitchen is an excellent environment for assessing human motor and cognitive function, as many complex actions are naturally exhibited in kitchens from chopping to cleaning. Here, we introduce the EPFL-Smart-Kitchen-30 dataset, collected in a noninvasive motion capture platform inside a kitchen environment. Nine static RGB-D cameras, inertial measurement units (IMUs) and one head-mounted HoloLens~2 headset were used to capture 3D hand, body, and eye movements. The EPFL-Smart-Kitchen-30 dataset is a multi-view action dataset with synchronized exocentric, egocentric, depth, IMUs, eye gaze, body and hand kinematics spanning 29.7 hours of 16 subjects cooking four different recipes. Action sequences were densely annotated with 33.78 action segments per minute. Leveraging this multi-modal dataset, we propose four benchmarks to advance behavior understanding and modeling through 1) a vision-language benchmark, 2) a semantic text-to-motion generation benchmark, 3) a multi-modal action recognition benchmark, 4) a pose-based action segmentation benchmark. We expect the EPFL-Smart-Kitchen-30 dataset to pave the way for better methods as well as insights to understand the nature of ecologically-valid human behavior. Code and data are available at https://github.com/amathislab/EPFL-Smart-Kitchen
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。