用多个小模型堆叠实现低成本通用语言能力
Stacking Small Language Models for Generalizability
- 将小模型分层堆叠，每层专注一个任务步骤
- 在多个基准上表现接近大模型，成本大幅降低
- 适合资源有限但需通用能力的场景
近期研究表明，大型语言模型（LLMs）在不同自然语言基准测试中表现出强大的泛化能力。然而，其庞大的规模导致训练和推理成本高昂，在资源受限环境下难以应用。本文提出一种新方法——微调小模型堆栈（FSLM），通过堆叠多个小型语言模型（SLMs）作为LLMs的替代方案。每个SLM经过微调后负责特定任务步骤，将高层次推理分解为多个低层次操作。该方法不仅显著降低训练与推理开销，还提升了模型可解释性，因各层模型通过自然语言交互。在常见自然语言基准上的评估表明，FSLM在保持较低成本的同时展现出有前景的通用性能，是替代大模型的可行路径。
原文摘要 · Abstract (English)
Recent advances show that large language models (LLMs) generalize strong performance across different natural language benchmarks. However, the large size of LLMs makes training and inference expensive and impractical to run in resource-limited settings. This paper introduces a new approach called fine-tuning stacks of language models (FSLM), which involves stacking small language models (SLM) as an alternative to LLMs. By fine-tuning each SLM to perform a specific task, this approach breaks down high level reasoning into multiple lower-level steps that specific SLMs are responsible for. As a result, FSLM allows for lower training and inference costs, and also improves model interpretability as each SLM communicates with the subsequent one through natural language. By evaluating FSLM on common natural language benchmarks, this paper highlights promising early results toward generalizable performance using FSLM as a cost-effective alternative to LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。