为大模型推理轨迹设计通用操作符,揭示其内在结构规律。
ReasonOps: Operator Segmentation for LLM Reasoning Traces

- 通过无监督聚类提取句子开头三词元,发现7种通用推理操作符。
- 操作符序列可准确识别模型来源,且能提前预测答案正确性。
- 适用于跨模型、跨任务的推理分析,适合研究者与开发者使用。
大型推理模型的思维链轨迹可达数万词元,但缺乏描述其内部结构的词汇体系。现有分析方法或过于僵化,或表达力不足,难以跨领域和模型捕捉特征。为此,我们提出ReasonOps,一种无监督、高表达力的思维链标注方法,提供简洁通用的操作符。利用ReasonOps,我们分析了来自12个思考型LLM(6大模型家族)在8个推理基准上的44,662条轨迹,发现它们共享一种组合式结构:7种重复出现的推理操作符(如回溯、推断、假设),由句子首三个词元的无监督聚类得出。这些操作符在所有模型家族和任务域中均出现,经三位独立的LLM裁判验证,对预留样本分类准确率达70%-76%。我们进一步分析简单与复杂问题中的操作符结构,发现反思类操作符在难题中更有效,在易题中反而降低性能。操作符序列具有强模型标识性:仅基于操作符分布的分类器即可以宏平均AUC识别源模型。结构特征显著提升答案正确性预测效果,分类器在WP-AUC和AIME上表现优于基线。此外,即使仅分析50%轨迹内容,也能实现WP-AUC的早期质量预估。整个流程无需人工标注,为深入理解LLM推理提供了新工具,并在模型识别与正确性预测上取得优异下游结果。
原文摘要 · Abstract (English)
Chain-of-thought traces from large reasoning models can span tens of thousands of tokens, yet we lack a vocabulary for describing their internal structure. Previous methods developed to analyze chain-of-thought traces are either too rigid or not expressive enough, failing to capture features across domains and models. To remedy this, we develop ReasonOps, an unsupervised, expressive method for annotating chain-of-thought traces, providing succinct universal operators. Using ReasonOps, we analyze 44,662 traces from 12 thinking LLMs spanning 6 families across 8 reasoning benchmarks and discover that they share a common compositional structure: 7 recurring reasoning operators -- discourse-level moves such as backtracking, inferring, and hypothesizing -- that emerge from unsupervised clustering of sentence-initial 3-token pivots. These operators appear across every model family and benchmark domain, confirmed by three independent LLM judges who classify held-out samples at 70 -76% accuracy. We analyze the structure of operators on easy vs. hard problems, revealing that reflective operators are more helpful on hard problems and harm performance on easy problems. Operator sequences are highly model-identifying: a classifier trained on operator distributions alone recovers the source model with macro-AUC, revealing that each model family has a distinctive reasoning fingerprint. Structural operator features predict within-problem answer correctness well above baselines. Classifiers built on these operators reach WP-AUC and on AIME specifically. ReasonOps further enables early quality estimation well before the trace completes: we predict at WP-AUC for only 50% of the trace. The ReasonOps pipeline is unsupervised and annotation-free, enabling deep insights into LLM reasoning traces as well as strong downstream results on model identification and correctness prediction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。