研究不同思维模式如何影响大模型表现,发现小模型适合结构化思考,大模型反而更适合自由发挥。
ThinkPatterns-21k: A Systematic Study on the Impact of Thinking Patterns in LLMs
- 构建21k条带五类思维模式的问答数据集,统一指令与答案只改内部思考方式。
- 300亿参数以下模型受益于结构化思考,320亿参数模型用分解思维反而表现下降。
- 无结构独白在各类模型中均有效,适合追求稳定性能的研究者使用。
大语言模型通过‘先思考再回答’的机制(即系统2思维)提升了性能,但现有研究缺乏对思维模式如何影响不同规模模型的系统性理解。本文开展全面分析,引入ThinkPatterns-21k数据集,包含21,000条来自现有指令跟随数据集的问答对,每对配有五种不同思维模式:一种无结构思考(独白)和四种结构化变体(分解、自问、自辩、自评),保持相同指令与响应。在3B至32B参数的不同模型上进行评估,发现:(1) 参数量小于300亿的模型普遍受益于结构化思维,而320亿参数模型采用分解等结构化思维会降低性能;(2) 无结构独白在各尺寸模型中均表现出广泛有效性。所有数据集、模型检查点及训练日志已公开,以支持可复现研究。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated enhanced performance through the \textit{Thinking then Responding} paradigm, where models generate internal thoughts before final responses (aka, System 2 thinking). However, existing research lacks a systematic understanding of the mechanisms underlying how thinking patterns affect performance across model sizes. In this work, we conduct a comprehensive analysis of the impact of various thinking types on model performance and introduce ThinkPatterns-21k, a curated dataset comprising 21k instruction-response pairs (QA) collected from existing instruction-following datasets with five thinking types. For each pair, we augment it with five distinct internal thinking patterns: one unstructured thinking (monologue) and four structured variants (decomposition, self-ask, self-debate and self-critic), while maintaining the same instruction and response. Through extensive evaluation across different model sizes (3B-32B parameters), we have two key findings: (1) smaller models (<30B parameters) can benefit from most of structured thinking patterns, while larger models (32B) with structured thinking like decomposition would degrade performance and (2) unstructured monologue demonstrates broad effectiveness across different model sizes. Finally, we released all of our datasets, checkpoints, training logs of diverse thinking patterns to reproducibility, aiming to facilitate further research in this direction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。