用最短路径问题测试大模型泛化能力，发现能迁移到新地图但难处理更长任务。
Generalization in LLM Problem Solving: The Case of the Shortest Path
- 构建可控的最短路径环境，分离训练数据、策略等影响因素。
- 模型在新地图上表现良好，但在更长路径任务中持续失败。
- 揭示推理时扩展无法弥补递归不稳定性，适合研究模型泛化机制者阅读。
语言模型能否系统性泛化仍存争议。然而，实证性能受训练数据、训练范式及推理策略等多重因素共同影响，导致失败原因难以解析。本文引入基于最短路径规划的可控合成环境，该问题为典型的可组合序列优化问题。该设置实现各因素的清晰分离，并支持两个正交的泛化维度：空间迁移至未见过的地图，以及长度扩展至更长轨迹的问题。实验发现，模型表现出强空间迁移能力，但在长度扩展下持续失败，根源在于递归不稳定性。进一步分析表明：数据覆盖范围设定了能力上限；强化学习虽提升训练稳定性，但未突破上限；推理时的缩放策略可提升性能，但无法挽救长度扩展失败。这些结果揭示了系统性求解能力的关键瓶颈。
原文摘要 · Abstract (English)
Whether language models can systematically generalize remains actively debated. Yet empirical performance is jointly shaped by multiple factors such as training data, training paradigms, and inference-time strategies, making failures difficult to interpret. We introduce a controlled synthetic environment based on shortest-path planning, a canonical composable sequential optimization problem. The setup enables clean separation of these factors and supports two orthogonal axes of generalization: spatial transfer to unseen maps and length scaling to longer-horizon problems. We find that models exhibit strong spatial transfer but consistently fail under length scaling due to recursive instability. We further analyze how distinct stages of the learning pipeline influence systematic problem-solving: for example, data coverage sets capability limits; reinforcement learning improves training stability but does not expand those limits; and inference-time scaling enhances performance but cannot rescue length-scaling failures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。