不改模型也能让大模型推理更快更省内存
ThinkLess: A Training-Free Inference-Efficient Method for Reducing Reasoning Redundancy
- 提前插入结束标记，跳过冗余推理步骤
- 推理时间减少70%以上，准确率几乎不变
- 无需训练，适合部署在资源受限场景
尽管思维链（CoT）提示能提升大语言模型的推理能力，但过长的推理序列会增加延迟和键值缓存占用，甚至在上下文长度限制下截断最终答案。我们提出ThinkLess，一种无需训练的推理优化框架，可在不修改模型的情况下提前终止推理生成并保持输出质量。注意力分析显示，答案标记对早期推理步骤关注极少，主要依赖推理结束标记，这是因果掩码下信息迁移的结果。基于此，ThinkLess将结束标记提前插入，跳过冗余推理同时保留知识传递。为防止提前终止导致格式混乱，采用轻量级后处理机制，利用模型自然的指令遵循能力生成结构化输出。无需微调或辅助数据，ThinkLess在保持与完整CoT相当准确率的同时，显著降低解码时间和内存消耗。
原文摘要 · Abstract (English)
While Chain-of-Thought (CoT) prompting improves reasoning in large language models (LLMs), the excessive length of reasoning tokens increases latency and KV cache memory usage, and may even truncate final answers under context limits. We propose ThinkLess, an inference-efficient framework that terminates reasoning generation early and maintains output quality without modifying the model. Atttention analysis reveals that answer tokens focus minimally on earlier reasoning steps and primarily attend to the reasoning terminator token, due to information migration under causal masking. Building on this insight, ThinkLess inserts the terminator token at earlier positions to skip redundant reasoning while preserving the underlying knowledge transfer. To prevent format discruption casued by early termination, ThinkLess employs a lightweight post-regulation mechanism, relying on the model's natural instruction-following ability to produce well-structured answers. Without fine-tuning or auxiliary data, ThinkLess achieves comparable accuracy to full-length CoT decoding while greatly reducing decoding time and memory consumption.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。