用双重稳健估计提升MCTS采样效率与决策质量
Doubly Robust Monte Carlo Tree Search
- 融合双重稳健估计算法与MCTS，结合模拟回溯与偏差修正
- 在井字棋中胜率88%（标准MCTS仅10%），虚拟家庭任务成功率20.7%（对比10.3%）
- 适合高样本成本场景，尤其适用于大模型下的高效决策
我们提出双重稳健蒙特卡洛树搜索（DR-MCTS），将双重稳健（DR）离策略估计引入蒙特卡洛树搜索（MCTS），以提升复杂环境中的样本效率与决策质量。该方法设计了一种混合估计器，结合MCTS模拟回溯与DR估计，在特定条件下提供无偏性与方差降低的理论保证。在井字棋和部分可观测的VirtualHome环境中进行的实验表明，DR-MCTS性能显著优于标准MCTS：井字棋中胜率达到88%，而标准MCTS仅为10%；在复合任务中，成功率为20.7%，相较标准MCTS的10.3%有明显提升。规模分析显示，当使用更大语言模型时，DR-MCTS仍保持更高样本效率，甚至在较小模型下也表现更优。这些结果凸显了其在样本受限的真实复杂场景中高效决策的潜力。
原文摘要 · Abstract (English)
We present Doubly Robust Monte Carlo Tree Search (DR-MCTS), a novel algorithm that integrates Doubly Robust (DR) off-policy estimation into Monte Carlo Tree Search (MCTS) to enhance sample efficiency and decision quality in complex environments. Our approach introduces a hybrid estimator that combines MCTS rollouts with DR estimation, offering theoretical guarantees of unbiasedness and variance reduction under specified conditions. Empirical evaluations in Tic-Tac-Toe and the partially observable VirtualHome environment demonstrate DR-MCTS's superior performance over standard MCTS. In Tic-Tac-Toe, DR-MCTS achieves an 88% win rate compared to a 10% win rate for standard MCTS. In compound VirtualHome tasks, DR-MCTS attains a 20.7% success rate versus 10.3% for standard MCTS. Our scaling analysis reveals that DR-MCTS exhibits better sample efficiency, notably outperforming standard MCTS with larger language models while using a smaller model. These results underscore DR-MCTS's potential for efficient decision-making in complex, real-world scenarios where sample efficiency is paramount.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。