双评论家比统一评论家更高效，让仿人机器人抓取快3.5倍。
Critic Architecture Matters: Dual vs. Unified Critics for Humanoid Loco-Manipulation
- 用独立评论家分别评估运动与操作目标，提升训练效率。
- 双评论家方案抓取速度达6.5步/次，是统一方案的3.5倍。
- 适合仿人机器人多任务强化学习的架构设计研究者参考。
针对仿人机器人多目标强化学习，需在单一策略中协调运动与操作。本文对比了统一评论家（单个估计综合价值）与双评论家（分离奖励信号）的设计。在NVIDIA Isaac Lab中，使用Unitree G1仿人机器人（23个自由度，17个由策略控制）进行分阶段课程训练，从静止抓取逐步过渡到带方向变化目标的行走抓取。在相同计算预算下，双评论家方案在标准评估中抓取速度达6.5步/次，较统一评论家的22.6步/次快3.5倍；抓取吞吐量达14.3次/1000步，为统一方案的2倍；验证抓取成功率也更高（65.2% vs. 53.8%）。即使额外添加五种反作弊奖励机制，双评论家效果未进一步提升（60.9% vs. 65.2%）。该差距可能源于训练策略、动作维度及奖励权重差异，但结果仍暗示：在模仿学习后微调强化学习时，统一评论家可能因运动梯度竞争抑制预训练手臂行为。因此，评论家架构应作为多目标仿人机器人强化学习的核心设计变量，并需通过单变量消融实验确认其因果贡献。代码、模型检查点及项目页见https://mturan33.github.io/critic-architecture-matters/
原文摘要 · Abstract (English)
Multi-objective reinforcement learning for humanoid robots must coordinate locomotion and manipulation within a single policy. A natural design choice is whether to use a single (unified) critic that estimates the combined value of all objectives, or separate (dual) critics with disjoint reward signals. We compare the two on the Unitree G1 humanoid (23 active DoF, of which 17 are policy-controlled) in NVIDIA Isaac Lab, training loco-manipulation policies through sequential curricula that progress from stationary reaching to walking with variable-orientation targets. Under a matched compute budget, the dual-critic run reaches targets 3.5x faster (6.5 vs. 22.6 simulation steps), achieves 2x higher throughput (14.3 vs. 7.0 validated reaches per 1,000 steps), and attains a higher validated reach rate (65.2% vs. 53.8%) than the unified-critic run in a standardized evaluation. Adding five anti-gaming reward mechanisms on top of the dual critic yields no further improvement (60.9% vs. 65.2%). We report this as an efficiency gap between two trained policies rather than an isolated effect of the critic: the two runs also differ in curriculum schedule, arm action dimensionality and one locomotion reward weight, and each is a single seed. The results are nonetheless suggestive for the emerging paradigm of RL fine-tuning of imitation-learned policies, where a unified critic may suppress pre-trained arm behavior through competing locomotion gradients. We argue that critic architecture deserves explicit treatment as a design variable in multi-objective humanoid RL, and specify the single-variable ablation required to establish its causal contribution. Code, trained checkpoints and a project page are available at https://mturan33.github.io/critic-architecture-matters/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。