用图结构分析大模型推理过程，发现模型思维模式比题目类型更关键。
ThinkProbe: Beyond Accuracy -- Structural Profiling of Open-Ended LLM Reasoning Traces via Non-Generative Thought Graphs
- 将推理步骤转为带循环的有向图，通过规则分割与语义链接构建认知图谱。
- 在4200条推理轨迹中发现模型级思维结构稳定，跨模型差异是跨领域差异的四倍。
- 揭示准确率无法捕捉的深层认知差异，适合研究模型内在思维机制者阅读。
我们提出ThinkProbe，一个用于大模型推理轨迹结构分析的框架。ThinkProbe将每条推理轨迹转化为包含8种节点类型和6种边类型的有向图（即思维图），并通过完全非生成式流程——结合规则分割与判别性语义链接——提取出19项指标构成的五维认知轮廓（5D-CP：广度、深度、结构、元认知、效率）。该方法应用于7个原生推理模型在200个开放问题和10个认知领域的4200条推理轨迹，结果表明：推理结构是稳定的模型级属性；在五个认知维度中的四个，模型间差异是领域间差异的四倍；仅‘结构’维度对问题领域敏感，暴露出准确率评估无法察觉的质性认知差异。
原文摘要 · Abstract (English)
We present ThinkProbe, a framework for structural analysis of LLM reasoning traces. ThinkProbe converts each trace into a Thought Graph a directed graph with cycles, 8 node types, and 6 edge types and derives a 19-metric five-dimensional cognitive profile (5D-CP: Breadth, Depth, Structure, Metacognitive, Efficiency) through a fully non-generative pipeline combining rule-based segmentation and discriminative semantic linking. Applied to 4{,}200 traces from 7 native reasoning models across 200 open-ended questions and 10 cognitive domains, ThinkProbe reveals that reasoning structure is a stable, model-level property: between-model variance exceeds between-domain variance by up to fourfold across four of five cognitive dimensions, with Structure showing genuine sensitivity to question domain, exposing qualitatively distinct cognitive profiles invisible to accuracy-based evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。