用光影重渲染技术,让机器人一次示范就能学会操作各种材质物体。
$\mathbf{M^3A}$ Policy: Mutable Material Manipulation Augmentation Policy through Photometric Re-rendering
- 通过光照传输特性进行光影重渲染,生成多样材质的虚拟示范
- 仅需一次真实示范,跨材质成功率平均提升58.03%
- 适合需要泛化到未知材质的机器人操控任务
材料泛化对现实世界机器人操作至关重要,尤其面对玻璃、金属等透明或反光表面带来的分布外变化。现有方法或依赖模拟环境进行仿真到现实迁移(受视觉域差距制约),或需大量真实示范(成本高且覆盖不足)。为此,本文提出可变材料操作增强框架M³A,利用计算摄影中的光传输物理特性进行光影重渲染。核心思路是:仅需一次真实示范,即可生成具有不同材质属性的高保真虚拟示范。该增强策略有效解耦了任务技能与表面外观,使策略无需额外数据收集即可跨材质泛化。为系统评估该能力,我们构建首个涵盖仿真与真实环境的多材质操作基准。大量实验表明,M³A策略显著提升跨材质泛化性能,在三项真实任务中平均成功率提高58.03%,并在未见过的材料上表现出鲁棒性。
原文摘要 · Abstract (English)
Material generalization is essential for real-world robotic manipulation, where robots must interact with objects exhibiting diverse visual and physical properties. This challenge is particularly pronounced for objects made of glass, metal, or other materials whose transparent or reflective surfaces introduce severe out-of-distribution variations. Existing approaches either rely on simulated materials in simulators and perform sim-to-real transfer, which is hindered by substantial visual domain gaps, or depend on collecting extensive real-world demonstrations, which is costly, time-consuming, and still insufficient to cover various materials. To overcome these limitations, we resort to computational photography and introduce Mutable Material Manipulation Augmentation (M$^3$A), a unified framework that leverages the physical characteristics of materials as captured by light transport for photometric re-rendering. The core idea is simple yet powerful: given a single real-world demonstration, we photometrically re-render the scene to generate a diverse set of highly realistic demonstrations with different material properties. This augmentation effectively decouples task-specific manipulation skills from surface appearance, enabling policies to generalize across materials without additional data collection. To systematically evaluate this capability, we construct the first comprehensive multi-material manipulation benchmark spanning both simulation and real-world environments. Extensive experiments show that the M$^3$A policy significantly enhances cross-material generalization, improving the average success rate across three real-world tasks by 58.03\%, and demonstrating robust performance on previously unseen materials.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。