用单条专家轨迹训练机器人，稳定高效地实现真实世界强化学习。
Off-Policy Actor-Critic with Sigmoid-Bounded Entropy for Real-World Robot Learning
- 引入符号函数约束熵，防止动作偏离分布导致优化失败。
- 在D4RL任务中实现100%成功率，且收敛速度优于已有方法。
- 仅需少量真实交互数据，适合资源受限的机器人部署场景。
真实世界中部署强化学习仍面临样本效率低、奖励稀疏和视觉观测噪声等问题。现有方法依赖大量演示数据或大规模预训练模型，成本高且易不稳定。本文提出SigEnt-SAC，一种从零开始训练的离线-在线强化学习方法，仅需一条专家轨迹即可学习。核心设计为符号函数约束熵项，可避免负熵驱动的分布外动作优化，并减少Q函数振荡。在D4RL基准上，该方法显著缓解了Q函数波动，更快达到100%成功率。进一步在四种不同机械臂上的真实机器人任务中验证，模型仅用少量真实交互与稀疏奖励，即可从原始图像中学习到有效策略，展现出低成本、实用化的现实部署潜力。
原文摘要 · Abstract (English)
Deploying reinforcement learning in the real world remains challenging due to sample inefficiency, sparse rewards, and noisy visual observations. Prior work leverages demonstrations and human feedback to improve learning efficiency and robustness. However, offline-to-online methods need large datasets and can be unstable, while VLA-assisted RL relies on large-scale pretraining and fine-tuning. As a result, a low-cost real-world RL method with minimal data requirements has yet to emerge. We introduce \textbf{SigEnt-SAC}, an off-policy actor-critic method that learns from scratch using a single expert trajectory. Our key design is a sigmoid-bounded entropy term that prevents negative-entropy-driven optimization toward out-of-distribution actions and reduces Q-function oscillations. We benchmark SigEnt-SAC on D4RL tasks against representative baselines. Experiments show that SigEnt-SAC substantially alleviates Q-function oscillations and reaches a 100\% success rate faster than prior methods. Finally, we validate SigEnt-SAC on four real-world robotic tasks across multiple embodiments, where agents learn from raw images and sparse rewards; results demonstrate that SigEnt-SAC can learn successful policies with only a small number of real-world interactions, suggesting a low-cost and practical pathway for real-world RL deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。