用思维树预测代码推理模型的正确率,提升低复杂度任务表现
Playing Psychic: Using Thought Trees to Predict Reasoning Models Accuracy on Coding Tasks

- 构建可生成任意难度代码任务的框架,覆盖真实场景
- 发现推理路径结构比内容更关键,正确率预测准确率达89%
- 轻量级分类器基于思维树特征,可自动识别异常路径并重试
大型语言模型在推理时扩展令牌预算,显著提升复杂任务表现,尤其在编程领域。然而现有评估多依赖竞赛类基准,难以反映真实推理能力。本文系统研究前沿推理模型在真实编程基准上的表现,提出一种程序化方法,从现有基准自动生成任意难度和结构的代码任务。分析表明,推理路径的结构本身是正确性的强预测因子,而非仅内容。为此,我们引入结构化思维树表示推理路径,并训练轻量分类器,通过提取思维树特征预测路径正确性。实验证明,基于该特征对结构异常路径进行标记并重试,可在低复杂度任务中持续获得性能提升。
原文摘要 · Abstract (English)
Recent advances in large language models (LLMs) have shown that test-time scaling can substantially improve model performance on complex tasks, particularly in the coding domain. Under this paradigm, models use a larger token budget during inference to generate intermediate reasoning traces before producing a final answer. However, current evaluations primarily rely on competitive programming benchmarks, which may not capture the full range of reasoning abilities. In this work, we perform a systematic study of frontier reasoning models to understand their performance on real-world coding benchmarks. To gain more insights into the performance of such models, we devise a programmatic way to {\em automatically generate} coding tasks of arbitrary difficulty and structure from existing benchmarks. Using this framework, our analysis reveals that the structure of a reasoning trace, not just its contents, is a strong predictor of correctness. Motivated by this, we propose structured thought-trees as means to represent reasoning traces. To illustrate their use, we train a lightweight classifier on features extracted from thought-trees to predict trace correctness, and demonstrate that flagging and retrying structurally anomalous traces based on the extracted features yields consistent gains at lower complexity levels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。