用可调控生成技术批量创建符合任务需求的3D场景。
Steerable Scene Generation with Post Training and Inference-Time Search
- 基于扩散模型生成物体位置与姿态，支持后续优化
- 通过强化学习与搜索实现目标导向场景合成，生成4400万+真实场景
- 适合机器人仿真、具身智能研究者使用
在仿真中训练机器人需要多样化的3D场景以反映实际任务挑战。然而，满足特定任务要求（如高杂乱度且空间布局合理）的场景稀缺且人工构建成本高昂。为此，我们采用程序化建模生成大规模场景数据，近似真实环境用于机器人操作，并将其适配至特定任务目标。方法基于统一的扩散生成模型，预测从固定资产库中选取的物体及其SE(3)位姿。该模型作为灵活的场景先验，可通过强化学习后训练、条件生成或推理时搜索进行调整，即使下游目标与原始数据分布不同也能有效引导生成。我们提出一种基于MCTS的扩散模型推理时搜索策略，结合投影与仿真确保物理可行性，并发布包含超过4400万组SE(3)位姿的场景数据集，覆盖五种不同环境。项目官网提供视频、代码、数据及模型权重。
原文摘要 · Abstract (English)
Training robots in simulation requires diverse 3D scenes that reflect the specific challenges of downstream tasks. However, scenes that satisfy strict task requirements, such as high-clutter environments with plausible spatial arrangement, are rare and costly to curate manually. Instead, we generate large-scale scene data using procedural models that approximate realistic environments for robotic manipulation, and adapt it to task-specific goals. We do this by training a unified diffusion-based generative model that predicts which objects to place from a fixed asset library, along with their SE(3) poses. This model serves as a flexible scene prior that can be adapted using reinforcement learning-based post training, conditional generation, or inference-time search, steering generation toward downstream objectives even when they differ from the original data distribution. Our method enables goal-directed scene synthesis that respects physical feasibility and scales across scene types. We introduce a novel MCTS-based inference-time search strategy for diffusion models, enforce feasibility via projection and simulation, and release a dataset of over 44 million SE(3) scenes spanning five diverse environments. Website with videos, code, data, and model weights: https://steerable-scene-generation.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。