用抽象符号代替长段推理,让模型更高效地思考。
Thinking Without Words: Efficient Latent Reasoning with Abstract Chain-of-Thought

- 用离散符号代替自然语言推理链,减少生成开销。
- 推理词数减少11.6倍,性能与传统方法相当。
- 适合追求高效推理的部署场景,通用性强。
虽然长而明确的思维链(CoT)在复杂推理任务中表现优异,但推理时生成成本高。非语言推理方法通过连续表示缩短生成长度,但性能较差。本文提出抽象思维链(Abstract-CoT),在语言模型中引入一个预留词汇表的离散潜在推理机制:模型先生成短序列符号,再生成答案。为使未见的抽象符号有效,设计了策略迭代式预热循环:(i) 通过掩码从自然语言思维链中瓶颈化并监督微调;(ii) 通过受限解码自蒸馏,仅凭提示生成抽象符号。预热后,采用冷启动强化学习优化抽象序列生成。实验表明,抽象-CoT 在数学推理、指令遵循和多跳推理任务上实现最多11.6倍的推理词数减少,性能相当,并跨不同语言模型家族泛化。我们还发现抽象词汇库呈现涌现的幂律分布,类似自然语言,且随训练阶段演化。结果表明,后训练的潜在推理机制可通过学习抽象推理语言实现高效推理。
原文摘要 · Abstract (English)
While long, explicit chains-of-thought (CoT) have proven effective on complex reasoning tasks, they are costly to generate during inference. Non-verbal reasoning methods have emerged with shorter generation lengths by leveraging continuous representations, yet their performance lags behind verbalized CoT. We propose $\textbf{Abstract Chain-of-Thought}$, a discrete latent reasoning post-training mechanism in which the language model produces a short sequence of tokens from a reserved vocabulary in lieu of a natural language CoT, before generating a response. To make previously unseen ''abstract'' tokens useful, we introduce a policy iteration-style warm-up loop that alternates between (i.) bottlenecking from a verbal CoT via masking and performing supervised fine-tuning, and (ii.) self-distillation by training the model to generate abstract tokens from the prompt alone via constrained decoding with the codebook. After warm-up, we optimize the generation of abstract sequences with warm-started reinforcement learning under constrained decoding. Abstract-CoT achieves up to $11.6\times$ fewer reasoning tokens while demonstrating comparable performance across mathematical reasoning, instruction-following, and multi-hop reasoning, and generalizes across language model families. We also find an emergent power law distribution over the abstract vocabulary, akin to those seen in natural language, that evolves across the training phases. Our findings highlight the potential for post-training latent reasoning mechanisms that enable efficient inference through a learned abstract reasoning language.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。