让大模型推理时跳过不重要的思考步骤，提速40%还不掉分。
TokenSkip: Controllable Chain-of-Thought Compression in LLMs
- 通过分析思考过程中的词元重要性，选择性跳过低贡献内容。
- 在GSM8K上减少40%思考词元（313→181），性能下降不足0.4%。
- 适合需要快速推理且对延迟敏感的应用场景。
思维链（CoT）已被证明能有效提升大语言模型（LLM）的推理能力。近期进展如OpenAI的o1和DeepSeek-R1表明，推理时延长思维链序列可进一步提升性能。然而，由于LLM解码的自回归特性，更长的思维链导致推理延迟线性增长，严重影响用户体验，尤其当思维链超过10,000词元时。为此，我们分析了思维链输出中词元的语义重要性，发现其对推理的贡献程度各异。基于此，提出TokenSkip方法，使模型可选择性跳过低重要性的词元，实现可控的思维链压缩。在多种模型与任务上的实验表明，TokenSkip能显著减少思维链词元使用量，同时保持强推理性能。以Qwen2.5-14B-Instruct为例，在GSM8K上推理词元从313降至181，减少40%，性能下降不足0.4%。代码与检查点已开源：https://github.com/hemingkx/TokenSkip。
原文摘要 · Abstract (English)
Chain-of-Thought (CoT) has been proven effective in enhancing the reasoning capabilities of large language models (LLMs). Recent advancements, such as OpenAI's o1 and DeepSeek-R1, suggest that scaling up the length of CoT sequences during inference could further boost LLM reasoning performance. However, due to the autoregressive nature of LLM decoding, longer CoT outputs lead to a linear increase in inference latency, adversely affecting user experience, particularly when the CoT exceeds 10,000 tokens. To address this limitation, we analyze the semantic importance of tokens within CoT outputs and reveal that their contributions to reasoning vary. Building on this insight, we propose TokenSkip, a simple yet effective approach that enables LLMs to selectively skip less important tokens, allowing for controllable CoT compression. Extensive experiments across various models and tasks demonstrate the effectiveness of TokenSkip in reducing CoT token usage while preserving strong reasoning performance. Notably, when applied to Qwen2.5-14B-Instruct, TokenSkip reduces reasoning tokens by 40% (from 313 to 181) on GSM8K, with less than a 0.4% performance drop. We release our code and checkpoints in https://github.com/hemingkx/TokenSkip.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。