arXiv:2601.14456cs.AIcs.LG2026-01被引 2

LLM规划模型在跨领域任务中失效,暴露其依赖表面模式而非真正推理能力。

On the Generalization Gap in LLM Planning: Tests and Verifier-Reward RL

  • 通过验证器奖励强化学习提升训练效率,但未改善跨域泛化
  • 在10个领域上微调后,域内有效计划率达82.9%,跨域为0%
  • 符号匿名化与序列压缩导致性能骤降,揭示对表层表示的敏感性

近期研究表明,微调后的大型语言模型(LLM)可在PDDL规划任务中实现高有效计划率。然而,这是否反映可迁移的规划能力,还是仅限于特定领域的记忆仍不明确。本研究在40,000个来自10个IPC 2023领域的域-问题-计划三元组上微调了一个17亿参数的LLM,评估其在域内和跨域下的泛化能力。尽管模型在域内条件下达到82.9%的有效计划率,但在两个未见过的域上表现归零。为分析失败原因,引入三项诊断干预:(i) 实例级符号匿名化,(ii) 紧凑计划序列化,(iii) 使用VAL验证器作为成功导向信号的验证器奖励强化学习。符号匿名化与紧凑序列化虽保持计划语义,却导致显著性能下降,表明模型对表面表示高度敏感。验证器奖励微调在半数监督训练周期内即达性能饱和,但未提升跨域泛化能力。在所探索配置下,域内性能稳定在约80%,而跨域性能崩溃,表明该微调模型严重依赖领域特定模式,而非可迁移的规划能力。结果凸显了基于LLM规划中的持续泛化差距,并提供了诊断工具以探究其成因。

原文摘要 · Abstract (English)

Recent work shows that fine-tuned Large Language Models (LLMs) can achieve high valid plan rates on PDDL planning tasks. However, it remains unclear whether this reflects transferable planning competence or domain-specific memorization. In this work, we fine-tune a 1.7B-parameter LLM on 40,000 domain-problem-plan tuples from 10 IPC 2023 domains, and evaluate both in-domain and cross-domain generalization. While the model reaches 82.9% valid plan rate in in-domain conditions, it achieves 0% on two unseen domains. To analyze this failure, we introduce three diagnostic interventions, namely (i) instance-wise symbol anonymization, (ii) compact plan serialization, and (iii) verifier-reward fine-tuning using the VAL validator as a success-focused reinforcement signal. Symbol anonymization and compact serialization cause significant performance drops despite preserving plan semantics, thus revealing strong sensitivity to surface representations. Verifier-reward fine-tuning reaches performance saturation in half the supervised training epochs, but does not improve cross-domain generalization. For the explored configurations, in-domain performance plateaus around 80%, while cross-domain performance collapses, suggesting that our fine-tuned model relies heavily on domain-specific patterns rather than transferable planning competence in this setting. Our results highlight a persistent generalization gap in LLM-based planning and provide diagnostic tools for studying its causes.

大模型规划泛化验证器

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。