不依赖模型蒸馏，用少量示例让大模型自动生成长链推理。
BOLT: Bootstrap Long Chain-of-Thought in Language Models without Distillation
- 用上下文学习从普通指令模型中自动构建长链推理数据
- 仅需10个示例即可启动，70B模型在多个基准上表现优异
- 适合想低成本提升模型推理能力的研究者和开发者
大型语言模型（如OpenAI的o1）展现出卓越的推理能力，其核心是生成长链思维（LongCoT），能够分析问题、制定计划、反思与回溯。然而，现有方法多依赖于从具备长链推理能力的模型（如o1、Qwen-QwQ、DeepSeek-R1-Preview）进行知识蒸馏，或依赖昂贵的人工标注，存在系统性不确定性且泛化性受限。本文提出BOLT（Bootstrap Long Chain-of-Thought without Distillation），无需蒸馏或人工标注，仅通过标准指令模型即可自举生成长链推理能力。该方法分三阶段：1）基于上下文学习，在标准指令模型上自举生成长链推理数据；2）监督微调以强化长链推理；3）在线训练持续优化。实验中仅需构造10个上下文示例，使用Llama-3.1-70B-Instruct作为基础模型，并扩展至7B、8B、70B等不同规模模型。在Arena-Hard、MT-Bench、WildBench、ZebraLogic、MATH500等多个评估基准上均取得显著性能提升，验证了方法的有效性与可扩展性。
原文摘要 · Abstract (English)
Large language models (LLMs), such as o1 from OpenAI, have demonstrated remarkable reasoning capabilities. o1 generates a long chain-of-thought (LongCoT) before answering a question. LongCoT allows LLMs to analyze problems, devise plans, reflect, and backtrack effectively. These actions empower LLM to solve complex problems. After the release of o1, many teams have attempted to replicate its LongCoT and reasoning capabilities. In terms of methods, they primarily rely on knowledge distillation with data from existing models with LongCoT capacities (e.g., OpenAI-o1, Qwen-QwQ, DeepSeek-R1-Preview), leaving significant uncertainties on systematically developing such reasoning abilities. In terms of data domains, these works focus narrowly on math while a few others include coding, limiting their generalizability. This paper introduces a novel approach to enable LLM's LongCoT capacity without distillation from o1-like models or expensive human annotations, where we bootstrap LongCoT (BOLT) from a standard instruct model. BOLT involves three stages: 1) LongCoT data bootstrapping with in-context learning on a standard instruct model; 2) LongCoT supervised finetuning; 3) online training to further refine LongCoT capacities. In BOLT, only a few in-context examples need to be constructed during the bootstrapping stage; in our experiments, we created 10 examples, demonstrating the feasibility of this approach. We use Llama-3.1-70B-Instruct to bootstrap LongCoT and apply our method to various model scales (7B, 8B, 70B). We achieve impressive performance on a variety of benchmarks, Arena-Hard, MT-Bench, WildBench, ZebraLogic, MATH500, which evaluate diverse task-solving and reasoning capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。