arXiv:2506.14162cs.LG2025-06

现有基准低估了程序化策略的泛化能力,简单调整神经网络训练即可实现相似效果。

Common Benchmarks Undervalue the Generalization Power of Programmatic Policies

  • 用更简单的神经网络架构和稀疏观测,让神经策略也能在分布外任务上泛化。
  • 采用更安全的奖励函数(如低速驾驶),可显著提升神经策略的泛化表现。
  • 建议设计需算法结构(如栈)的任务来真正区分程序化与神经策略的能力。

用于序列决策问题的学习程序化表示方法常在分布外(OOD)任务上评估,普遍认为程序化策略比神经策略具有更强的泛化能力。本文指出,现有基准低估了程序化表示的泛化潜力。我们分析了四篇文献中的实验,发现原本被认为无法泛化的神经策略,通过训练管道的简单调整——如使用相同类型的稀疏观测、更简单的神经网络架构——即可达到与程序化策略相当的OOD泛化效果。另一有效改进是采用促进安全性的奖励函数(例如鼓励缓慢驾驶的策略)。此外,我们主张构建强调算法性概念(如栈)的基准任务,以真正检验不同策略在复杂逻辑推理上的差异。

原文摘要 · Abstract (English)

Algorithms for learning programmatic representations for sequential decision-making problems are often evaluated on out-of-distribution (OOD) problems, with the common conclusion that programmatic policies generalize better than neural policies on OOD problems. In this position paper, we argue that commonly used benchmarks undervalue the generalization capabilities of programmatic representations. We analyze the experiments of four papers from the literature and show that neural policies, which were shown not to generalize, can generalize as effectively as programmatic policies on OOD problems. This is achieved with simple changes in the neural policies training pipeline. Namely, we show that simpler neural architectures with the same type of sparse observation used with programmatic policies can help attain OOD generalization. Another modification we have shown to be effective is the use of reward functions that allow for safer policies (e.g., agents that drive slowly can generalize better). Also, we argue for creating benchmark problems highlighting concepts needed for OOD generalization that may challenge neural policies but align with programmatic representations, such as tasks requiring algorithmic constructs like stacks.

程序化策略泛化能力强化学习基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。