思考过程反而降低模型表现，尤其在人类也因深思而犯错的任务上
Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse
- 借鉴认知心理学，测试六类人类深思会出错的任务
- 三类任务中模型准确率下降最高达36.3%（o1-preview比GPT-4o差）
- 提醒研究者：不是所有任务都适合用思维链推理
思维链（Chain-of-thought, CoT）提示已成为提升大语言和多模态模型性能的常用策略。然而，在哪些场景下CoT会系统性降低性能仍是未解之谜。本文受认知心理学启发，聚焦心理文献中六类人类深思会降低表现的代表性任务，发现其中三类任务中，当前最先进模型使用CoT后出现显著性能下降（OpenAI o1-preview相比GPT-4o绝对准确率下降高达36.3%），其余任务则效果混合，有正有负。尽管模型与人类认知过程不完全一致，但人类在思考时出错的场景可为识别模型受负面影响的条件提供线索。通过连接人类言语思维与模型推理评估，本文为理解推理时思维链的影响提供了新视角。
原文摘要 · Abstract (English)
Chain-of-thought (CoT) prompting has become a widely used strategy for improving large language and multimodal model performance. However, it is still an open question under which settings CoT systematically reduces performance. In this paper, we seek to identify the characteristics of tasks where CoT reduces performance by drawing inspiration from cognitive psychology, focusing on six representative tasks from the psychological literature where deliberation hurts performance in humans. In three of these tasks, state-of-the-art models exhibit significant performance drop-offs with CoT (up to 36.3\% absolute accuracy for OpenAI o1-preview compared to GPT-4o), while in others, CoT effects are mixed, with positive, neutral, and negative changes. While models and humans do not exhibit perfectly parallel cognitive processes, considering cases where thinking has negative consequences for humans helps identify settings where it negatively impacts models. By connecting the literature on human verbal thinking and deliberation with evaluations of CoT, we offer a perspective for understanding the impact of inference-time reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。