让小模型学会像大模型一样有策略地思考。
FutureMind: Equipping Small Language Models with Strategic Thinking-Pattern Priors via Adaptive Knowledge Distillation
- 通过自适应知识蒸馏,将大模型的思维模式迁移到小模型。
- 在多跳问答任务上超越强基线,实现小模型的顶尖表现。
- 适合追求高效推理又需复杂推理能力的场景使用。
小型语言模型(SLMs)因推理高效、延迟低,适用于资源受限场景。然而,在需要结构化推理与有效检索的复杂知识任务中表现不佳。为此,我们提出FutureMind,一个模块化推理框架,通过从大语言模型(LLMs)中自适应地进行知识蒸馏,赋予SLMs战略思维模式先验。FutureMind采用动态推理流程,包含问题分析、逻辑推理、策略规划和检索引导四个核心模块,并集成三种检索范式,将复杂查询分解为可处理的子问题,确保高效准确的检索执行。在2WikiMultihopQA、MuSiQue、Bamboogle和Frames等多跳问答基准上,FutureMind持续优于Search-o1等强基线,在不同SLM架构与规模下均达到自由训练条件下的最先进水平。此外,分析发现教师(LLMs)与学生(SLMs)之间的认知偏见瓶颈限制了思维模式迁移,为推理技能可迁移性提供了新视角,推动兼具效率与真实认知能力的小模型发展。
原文摘要 · Abstract (English)
Small Language Models (SLMs) are attractive for cost-sensitive and resource-limited settings due to their efficient, low-latency inference. However, they often struggle with complex, knowledge-intensive tasks that require structured reasoning and effective retrieval. To address these limitations, we propose FutureMind, a modular reasoning framework that equips SLMs with strategic thinking-pattern priors via adaptive knowledge distillation from large language models (LLMs). FutureMind introduces a dynamic reasoning pipeline composed of four key modules: Problem Analysis, Logical Reasoning, Strategy Planning, and Retrieval Guidance. This pipeline is augmented by three distinct retrieval paradigms that decompose complex queries into tractable subproblems, ensuring efficient and accurate retrieval execution. Extensive experiments on multi-hop QA benchmarks, including 2WikiMultihopQA, MuSiQue, Bamboogle, and Frames, demonstrate the superiority of FutureMind. It consistently outperforms strong baselines such as Search-o1, achieving state-of-the-art results under free training conditions across diverse SLM architectures and scales. Beyond empirical gains, our analysis reveals that the process of thinking-pattern distillation is restricted by the cognitive bias bottleneck between the teacher (LLMs) and student (SLMs) models. This provides new perspectives on the transferability of reasoning skills, paving the way for the development of SLMs that combine efficiency with genuine cognitive capability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。