arXiv:2504.16595cs.ROcs.LG2025-04

用人类示范提升强化学习,高效打包不规则物体。

HERB: Human-augmented Efficient Reinforcement learning for Bin-packing

  • 融合人类示范与强化学习,自动学习打包策略。
  • 打包效率和延迟均优于传统方法,稳定性更好。
  • 适合机器人仓储、物流等需要人机协作的场景。

高效打包物品是物流、仓储自动化和机器人领域的基础问题。面对高度多样的3D物体(如家居或生鲜商品),闭式解不可行,而传统的启发式或无模型强化学习方法仅关注几何优化,依赖对离散解空间的穷举搜索,导致训练时间长(纯强化学习)或延迟高(启发式),且难以迁移到机器人场景,忽略物体特性(易碎性、可变形性)和人类偏好。我们提出HERB,首个利用人类示范解决不规则物体打包问题的人机协同强化学习框架。该框架借助人类专家在打包策略中的示范数据,其中蕴含了空间优化、稳定性及物体属性等隐含因素,这些因素难以显式建模。通过将人类示范与强化学习探索相结合,实现容器内每件物品的精准放置。实验表明,本方法在打包效率和延迟上均优于启发式、纯强化学习及模仿学习方法。定性结果表明,生成的打包方案更稳定、更接近人类习惯,预计更具可接受性和应用价值。最后,我们在机器人系统上验证了该方法的实际可行性。

原文摘要 · Abstract (English)

Packing objects efficiently is a fundamental problem in logistics, warehouse automation, and robotics. When dealing with highly diverse 3D objects (household or grocery items), closed-form solutions are infeasible, and heuristic or model-free Reinforcement Learning~(RL) methods tend to focus solely on geometric optimization, relying on exhaustive searches of the discretized solution space. This leads to long training times (for pure RL) and high latency (heuristics), limited transferability to robotic scenarios, and ultimately ignores object characteristics (fragility, deformability) and human preferences. We propose HERB, a human-augmented RL framework for packing irregular objects, the first to explore the potential of learning from human demonstrations to solve this complex task. It leverages human demonstrations of packing strategies, which inherently exhibit latent factors such as space optimization, stability, and object properties that are difficult to model explicitly. The human-expert data is combined with RL exploration to provide the placement of each object inside the container. Experimental results show that our method outperforms heuristic, purely RL-based, and imitation learning approaches in packing efficiency and latency. Qualitative results highlight that our packing strategy produces more stable, human-like arrangements, which we expect to be more appropriate and widely accepted. Finally, we demonstrate the real-world feasibility of our method on a robotic system.

强化学习机器人打包优化人机协同

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。