arXiv:2410.07500cs.CV2024-10ICLR被引 6

从带噪网络视频中学习多样行人运动,生成更真实的城市行进轨迹。

Learning to Generate Diverse Pedestrian Movements from Web Videos with Noisy Labels

  • 用自动标签过滤和掩码嵌入处理网络视频的噪声标签。
  • 在城市场景下生成多样化、符合上下文的行人运动轨迹,零样本泛化能力出色。
  • 适合做交通模拟、自动驾驶行为预测的研究者使用。

理解并建模真实世界中的行人运动对运动预测和场景仿真至关重要。现有方法常忽略场景上下文、个体特征与目标等影响因素。网络视频包含自然行人行为和丰富运动上下文,但用预训练模型标注会引入噪声标签。本文构建了大规模数据集CityWalkers,捕捉城市环境中多样的行人运动。基于此,提出生成模型PedGen,通过自动标签过滤去除低质量标签,并采用掩码嵌入支持部分标签训练。其创新的上下文编码器将2D场景升维至3D,融合多种上下文因素,生成更真实的行人运动。实验表明,PedGen在噪声标签下仍优于现有基线方法,且在真实与仿真环境均实现零样本泛化。代码、模型与数据将公开于https://genforce.github.io/PedGen/。

原文摘要 · Abstract (English)

Understanding and modeling pedestrian movements in the real world is crucial for applications like motion forecasting and scene simulation. Many factors influence pedestrian movements, such as scene context, individual characteristics, and goals, which are often ignored by the existing human generation methods. Web videos contain natural pedestrian behavior and rich motion context, but annotating them with pre-trained predictors leads to noisy labels. In this work, we propose learning diverse pedestrian movements from web videos. We first curate a large-scale dataset called CityWalkers that captures diverse real-world pedestrian movements in urban scenes. Then, based on CityWalkers, we propose a generative model called PedGen for diverse pedestrian movement generation. PedGen introduces automatic label filtering to remove the low-quality labels and a mask embedding to train with partial labels. It also contains a novel context encoder that lifts the 2D scene context to 3D and can incorporate various context factors in generating realistic pedestrian movements in urban scenes. Experiments show that PedGen outperforms existing baseline methods for pedestrian movement generation by learning from noisy labels and incorporating the context factors. In addition, PedGen achieves zero-shot generalization in both real-world and simulated environments. The code, model, and data will be made publicly available at https://genforce.github.io/PedGen/ .

行人生成城市模拟噪声标签零样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。