用知识空间理论检验大模型数学推理是否具备人类般的知识结构
Do LLMs Exhibit Coherent Knowledge Structures in Mathematical Reasoning? A Perspective from Knowledge Space Theory
- 基于知识空间理论构建评估框架，分析模型对知识点依赖关系的理解
- 模型常违反知识依赖，且无法利用上下文相关知识提升表现
- 不同模型知识结构差异大，现有评测方法难以发现这些缺陷
人类知识具有内在结构和依赖关系：掌握某一概念需先掌握其前置知识，这一原理由知识空间理论（KST）形式化。尽管大模型在复杂推理任务中表现优异，但其是否具备类似人类的知识结构仍不明确。本文提出一种基于KST的评估框架，用于分析大模型在数学推理中的知识结构，考察其行为是否遵循合理的知识依赖关系。对比八种开源与闭源大模型与真实人类学习者的表现，结果表明：（1）大模型不遵循人类知识结构——它们频繁违反知识依赖，且无法利用上下文提供的相关知识提升对依赖性问题的解答性能；（2）模型间知识分布重叠度低，缺乏一致的知识结构。此外，这些结构性缺陷在基于准确率和大模型自评的评估中几乎不可见。研究为当前大模型知识不具备人类式结构提供了行为证据。
原文摘要 · Abstract (English)
Human knowledge is inherently structured and interdependent: mastery of a concept requires prior mastery of its prerequisites, a principle formalized by Knowledge Space Theory (KST). While LLMs achieve strong performance on complex reasoning tasks, it remains unclear whether they exhibit coherent, human-like knowledge structure. We introduce a KST-grounded framework for evaluating LLM knowledge structure in mathematical reasoning, using it as a normative framework to analyze whether LLM behavior adheres to principled knowledge dependencies. Evaluating eight open- and closed-source LLMs against real human learners, we find that (1) LLMs do not adhere to human knowledge structure -- they frequently violate knowledge dependencies and fail to leverage related knowledge provided in context to improve performance on dependent questions; (2) LLMs do not share a consistent knowledge structure among themselves, as reflected by low overlap in their knowledge distributions. Furthermore, these structural deficiencies remain largely invisible to accuracy-based and LLM-as-judge evaluations. Together, our results provide behavioral evidence that current LLMs knowledge does not follow a human-like structure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。