用图结构分析大模型推理过程,揭示提示策略如何影响推理质量。
Mapping the Minds of LLMs: A Graph-Based Analysis of Reasoning LLM
- 将长篇思维链聚类为语义连贯步骤,构建有向推理图。
- 发现探索密度、分叉度等结构特征与推理准确率强相关。
- 适合研究模型推理机制或优化提示工程的读者。
近期测试时扩展技术使大语言模型(LLMs)通过生成长链式思维(CoT)展现出复杂推理能力。然而,这些推理型大模型(RLMs)常表现出反直觉且不稳定的特性,例如在少样本提示下性能下降,挑战了当前对RLMs的理解。本文提出一种统一的基于图的分析框架,用于更准确建模RLMs的推理过程。方法首先将冗长的CoT输出聚类为语义一致的推理步骤,再构建有向推理图以捕捉步骤间的上下文与逻辑依赖关系。在多种模型和提示策略下进行综合分析,发现结构性指标如探索密度、分叉程度和收敛比与推理准确性显著相关。结果表明,提示策略会显著重塑RLMs内部的推理结构,直接影响任务表现。该框架不仅实现了超越传统指标的推理质量量化评估,还为提示工程与大模型认知分析提供了实用洞见。代码与资源将公开,以促进后续研究。
原文摘要 · Abstract (English)
Recent advances in test-time scaling have enabled Large Language Models (LLMs) to display sophisticated reasoning abilities via extended Chain-of-Thought (CoT) generation. Despite their potential, these Reasoning LLMs (RLMs) often demonstrate counterintuitive and unstable behaviors, such as performance degradation under few-shot prompting, that challenge our current understanding of RLMs. In this work, we introduce a unified graph-based analytical framework for better modeling the reasoning processes of RLMs. Our method first clusters long, verbose CoT outputs into semantically coherent reasoning steps, then constructs directed reasoning graphs to capture contextual and logical dependencies among these steps. Through comprehensive analysis across models and prompting regimes, we reveal that structural properties, such as exploration density, branching, and convergence ratios, strongly correlate with reasoning accuracy. Our findings demonstrate how prompting strategies substantially reshape the internal reasoning structure of RLMs, directly affecting task outcomes. The proposed framework not only enables quantitative evaluation of reasoning quality beyond conventional metrics but also provides practical insights for prompt engineering and the cognitive analysis of LLMs. Code and resources will be released to facilitate future research in this direction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。