测试大模型在真实知识图谱中规划路径的能力,发现顶尖模型仍有明显短板。
LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs?
- 通过跳转维基链接从起点到目标页,考察模型的长程规划与常识推理能力。
- 最强模型Gemini-3在困难任务中仅23%成功,表明当前模型难以应对复杂路径。
- 模型失败后常陷入循环,缺乏有效重规划能力,适合研究推理机制的学者参考。
我们提出LLM-Wikirace基准,用于评估大语言模型(LLMs)在规划、推理和世界知识方面的能力。该任务要求模型通过逐步点击维基百科超链接,从起始页面到达目标页面,需具备前瞻规划能力及对现实世界概念关联的理解。我们评估了多种开源与闭源模型,包括Gemini-3、GPT-5和Claude Opus 4.5,它们在简单任务中表现优异,甚至达到超人水平。然而,在困难任务上性能急剧下降:最强模型Gemini-3仅在23%的难题中成功,暴露出前沿模型仍面临巨大挑战。分析显示,世界知识虽关键,但超过一定阈值后,规划与长程推理能力成为决定性因素。轨迹级分析进一步揭示,即使最强模型在失败后也难以重新规划,频繁陷入循环。该基准简洁却能清晰暴露当前推理系统的局限,为具备规划能力的LLMs提供了开放的验证平台。代码与排行榜见https://llmwikirace.github.io。
原文摘要 · Abstract (English)
We introduce LLM-Wikirace, a benchmark for evaluating planning, reasoning, and world knowledge in large language models (LLMs). In LLM-Wikirace, models must efficiently navigate Wikipedia hyperlinks step by step to reach a target page from a given source, requiring look-ahead planning and the ability to reason about how concepts are connected in the real world. We evaluate a broad set of open- and closed-source models, including Gemini-3, GPT-5, and Claude Opus 4.5, which achieve the strongest results on the easy level of the task and demonstrate superhuman performance. Despite this, performance drops sharply on hard difficulty: the best-performing model, Gemini-3, succeeds in only 23\% of hard games, highlighting substantial remaining challenges for frontier models. Our analysis shows that world knowledge is a necessary ingredient for success, but only up to a point, beyond this threshold, planning and long-horizon reasoning capabilities become the dominant factors. Trajectory-level analysis further reveals that even the strongest models struggle to replan after failure, frequently entering loops rather than recovering. LLM-Wikirace is a simple benchmark that reveals clear limitations in current reasoning systems, offering an open arena where planning-capable LLMs still have much to prove. Our code and leaderboard available at https:/llmwikirace.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。