让大模型推理更简洁高效，自动删减冗余思考过程
Structured Thoughts For Improved Reasoning And Context Pruning
- 将推理分为试错与结论两部分，结构化输出
- 比普通微调提升8.08%推理准确率，节省85%上下文内存
- 适合需要高效推理的部署场景，如移动端或实时系统
大型语言模型在生成长链思维时表现优异，但推理过程冗长且内存效率低。本文提出结构化思维框架，将推理分为交替的<try>（探索性草稿）和<outcome>（结论提炼）块。通过分割推理轨迹并用LLM生成对应<outcome>，构建结构化思维数据集。在该数据上微调预训练模型，可使模型采用结构化推理风格，在推理基准测试中性能最高提升8.08%。显式结构支持上下文剪枝：每对<try>/<outcome>后可删除<try>，仅保留结论，实现平均85%的内存/上下文节省，数学任务性能下降仅8.67%。
原文摘要 · Abstract (English)
Large language models (LLMs) excel at generating long chains of thought, but long reasoning traces are often verbose and memory-inefficient. In this work, we introduce Structured Thoughts, a framework that organizes reasoning into alternating <try> and <outcome> blocks: <try> captures exploratory scratch work, while <outcome> contains the distilled conclusion of that step. We construct a dataset of structured thoughts by segmenting reasoning traces into <try> blocks and prompting an LLM to summarize each step into its corresponding <outcome>. Fine-tuning pretrained foundation models on this reformatted data produces models that adopt the structured reasoning style, leading to performance gains of up to 8.08\% on reasoning benchmarks compared to standard SFT. The explicit structure also enables context pruning: after each <try>/<outcome> pair, the <try> can be pruned, allowing the model to retain conclusions without keeping the full scratch work in the context. A proof-of-concept pruning implementation achieves an average of 85\% memory / context savings with an 8.67\% performance drop across mathematical tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。