arXiv:2501.07641cs.CL2025-01被引 1

将GPT视为概率树,揭示其推理本质是模式匹配而非逻辑推演。

GPT as a Monte Carlo Language Tree: A Probabilistic Perspective

  • 用蒙特卡洛语言树建模数据与GPT,节点为词元,边为转移概率。
  • 87%以上GPT输出可由数据树回溯,大模型更接近数据树结构。
  • 解释幻觉、思维链等现象,适合研究模型机理的学者参考。

大型语言模型(如GPT)通过预测下一个词元来完成自然语言处理任务,但其隐式分布建模机制缺乏量化理解。本文提出新视角:任何语言数据集可表示为蒙特卡洛语言树(Data-Tree),每个节点代表一个词元,每条边代表词元转移概率,每条序列对应唯一路径。类似地,GPT类模型也可被展开为另一个蒙特卡洛语言树(GPT-Tree)。实验显示,同一数据集上训练的不同GPT模型在GPT-Tree可视化中具有显著结构相似性,且模型越大越趋近于Data-Tree。超过87%的GPT输出词元可被Data-Tree回溯。这些发现表明LLM的推理过程更可能是基于最大概率的模式匹配,而非形式化推理。此外,本文深入分析了幻觉、思维链(CoT)推理和词元偏差等问题。

原文摘要 · Abstract (English)

Large Language Models (LLMs), such as GPT, are considered to learn the latent distributions within large-scale web-crawl datasets and accomplish natural language processing (NLP) tasks by predicting the next token. However, this mechanism of latent distribution modeling lacks quantitative understanding and analysis. In this paper, we propose a novel perspective that any language dataset can be represented by a Monte Carlo Language Tree (abbreviated as ``Data-Tree''), where each node denotes a token, each edge denotes a token transition probability, and each sequence has a unique path. Any GPT-like language model can also be flattened into another Monte Carlo Language Tree (abbreviated as ``GPT-Tree''). Our experiments show that different GPT models trained on the same dataset exhibit significant structural similarity in GPT-Tree visualization, and larger models converge more closely to the Data-Tree. More than 87\% GPT output tokens can be recalled by Data-Tree. These findings may confirm that the reasoning process of LLMs is more likely to be probabilistic pattern-matching rather than formal reasoning, as each model inference seems to find a context pattern with maximum probability from the Data-Tree. Furthermore, we provide deeper insights into issues such as hallucination, Chain-of-Thought (CoT) reasoning, and token bias in LLMs.

大模型机理概率建模文本生成模型解释

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。