用压缩的思维胶囊提升大模型推理效率,兼顾速度与可解释性。
R-Capsule: Compressing High-Level Plans for Efficient Large Language Model Reasoning
- 将复杂推理步骤压缩为少量学习过的隐式令牌,实现高效推理。
- 在多个基准测试上,推理长度减少60%以上,准确率反而提升1-3%。
- 适合需要低延迟、高透明度推理的应用场景,如智能客服与自动决策。
链式思维(CoT)提示能帮助大语言模型处理复杂推理任务,但其冗长的中间步骤会增加延迟和内存消耗,并可能传播早期错误。本文提出推理胶囊(R-Capsule)框架,结合隐式推理的效率与显式CoT的可解释性。核心思想是将高层推理计划压缩为一组少量学习的隐式令牌(即推理胶囊),同时保持执行步骤轻量或显式。该方法受信息瓶颈(IB)原理启发,通过低容量瓶颈增强最小性以提升效率,辅以主任务损失与辅助计划重构损失共同保证充分性。重构损失使隐空间有实际意义,减少无用捷径使用。实验表明,该框架在降低60%以上可见令牌数量的同时,维持甚至提升了复杂基准上的准确率。代码已公开于 https://anonymous.4open.science/r/Reasoning-Capsule-7BE0。
原文摘要 · Abstract (English)
Chain-of-Thought (CoT) prompting helps Large Language Models (LLMs) tackle complex reasoning by eliciting explicit step-by-step rationales. However, CoT's verbosity increases latency and memory usage and may propagate early errors across long chains. We propose the Reasoning Capsule (R-Capsule), a framework that aims to combine the efficiency of latent reasoning with the transparency of explicit CoT. The core idea is to compress the high-level plan into a small set of learned latent tokens (a Reasoning Capsule) while keeping execution steps lightweight or explicit. This hybrid approach is inspired by the Information Bottleneck (IB) principle, where we encourage the capsule to be approximately minimal yet sufficient for the task. Minimality is encouraged via a low-capacity bottleneck, which helps improve efficiency. Sufficiency is encouraged via a dual objective: a primary task loss for answer accuracy and an auxiliary plan-reconstruction loss that encourages the capsule to faithfully represent the original textual plan. The reconstruction objective helps ground the latent space, thereby improving interpretability and reducing the use of uninformative shortcuts. Our framework strikes a balance between efficiency, accuracy, and interpretability, thereby reducing the visible token footprint of reasoning while maintaining or improving accuracy on complex benchmarks. Our codes are available at: https://anonymous.4open.science/r/Reasoning-Capsule-7BE0
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。