研究发现:训练时用低效推理路径,反而让模型在新问题上表现更好。
On the Bias of Next-Token Predictors Toward Systematically Inefficient Reasoning: A Shortest-Path Case Study
- 用带回溯的长路径训练模型,比用最优路径效果更好
- 相同训练量下,冗余路径使模型在新图上准确率更高
- 模型对下一步预测越有信心,泛化能力越强
近期自然语言处理进展表明,提升大语言模型推理能力的关键在于:(i) 增加测试时计算有助于解决难题,但常引入推理轨迹冗余;(ii) 当推理具有系统性与增量性时,计算效率最高,类似人类逐步思考的链式思维(CoTs)。为独立研究这两个因素,我们基于分层图上的最短路径任务构建了受控实验。使用自定义分词器,训练解码器仅的Transformer模型,对比基于最优自底向上动态规划路径与包含回溯的更长有效路径的训练结果。令人意外的是,在相同训练令牌预算下,使用低效路径训练的模型在未见图上泛化性能更优。该优势并非单纯由路径长度导致——随意插入冗余信息不仅无益,甚至损害性能。相反,我们发现泛化能力与模型对下一个词预测的信心正相关,表明长而连贯、局部增量的推理轨迹使训练信号更易优化。
原文摘要 · Abstract (English)
Recent advances in natural language processing highlight two key factors for improving reasoning in large language models (LLMs): (i) allocating more test-time compute tends to help on harder problems but often introduces redundancy in the reasoning trace, and (ii) compute is most effective when reasoning is systematic and incremental, forming structured chains of thought (CoTs) akin to human problem-solving. To study these factors in isolation, we introduce a controlled setting based on shortest-path tasks in layered graphs. We train decoder-only transformers on question-trace-answer triples using a custom tokenizer, comparing models trained on optimal bottom-up dynamic programming traces with those trained on longer, valid traces involving backtracking. Surprisingly, with the same training-token budget, models trained on inefficient traces generalize better to unseen graphs. This benefit is not due to length alone-injecting arbitrary redundancy into reasoning traces fails to help and can even hurt performance. Instead, we find that generalization correlates with the model's confidence in next-token prediction, suggesting that long, coherent, and locally incremental traces make the training signal easier to optimize.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。