对比三模型在复杂关系推理中的表现,发现大模型仍有局限。
Are Large Language Models Capable of Deep Relational Reasoning? Insights from DeepSeek-R1 and Benchmark Comparisons
- 设计家族树与图推理任务测试模型逻辑推断能力
- DeepSeek-R1在多任务中表现最优,但复杂度上升时仍失败
- 揭示模型推理过程中的规划与验证机制,也暴露不连贯问题
大型语言模型(LLMs)在深层关系推理方面有多强?本文通过精心设计的家族树和通用图推理基准任务,评估并比较了三款前沿LLM——DeepSeek-R1、DeepSeek-V3和GPT-4o的推理能力。实验表明,DeepSeek-R1在多个任务和问题规模下均取得最高F1分数,展现出较强的逻辑演绎与关系推断能力。然而,所有模型,包括DeepSeek-R1,随着问题复杂度提升而显著退化,主要受限于上下文长度及输出结构不完整。对DeepSeek-R1长链思维过程的详细分析揭示其独特的规划与验证策略,但也暴露出推理不连贯或不完整的情况,凸显需深入审视模型内部推理动态。本文进一步探讨未来方向,包括多模态推理的作用及推理失败的系统性分析。研究结果为提升LLMs在结构化、多步逻辑推理任务中的表现提供了实证洞察与理论启示。代码库将公开于https://github.com/kelvinhkcs/Deep-Relational-Reasoning。
原文摘要 · Abstract (English)
How far are Large Language Models (LLMs) in performing deep relational reasoning? In this paper, we evaluate and compare the reasoning capabilities of three cutting-edge LLMs, namely, DeepSeek-R1, DeepSeek-V3 and GPT-4o, through a suite of carefully designed benchmark tasks in family tree and general graph reasoning. Our experiments reveal that DeepSeek-R1 consistently achieves the highest F1-scores across multiple tasks and problem sizes, demonstrating strong aptitude in logical deduction and relational inference. However, all evaluated models, including DeepSeek-R1, struggle significantly as problem complexity increases, largely due to token length limitations and incomplete output structures. A detailed analysis of DeepSeek-R1's long Chain-of-Thought responses uncovers its unique planning and verification strategies, but also highlights instances of incoherent or incomplete reasoning, calling attention to the need for deeper scrutiny into LLMs' internal inference dynamics. We further discuss key directions for future work, including the role of multimodal reasoning and the systematic examination of reasoning failures. Our findings provide both empirical insights and theoretical implications for advancing LLMs' reasoning abilities, particularly in tasks that demand structured, multi-step logical inference. Our code repository will be publicly available at https://github.com/kelvinhkcs/Deep-Relational-Reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。