无需外部帮助，自动生成高质量题目让大模型自我提升。
The Self-Improvement Paradox: Can Language Models Bootstrap Reasoning Capabilities without External Scaffolding?
- 用提示词诱导生成问题，再通过拒收采样去重，最后用多数投票获取答案。
- 仅用自生成数据就能提升模型数学推理能力，且零样本性能不下降。
- 适合想实现无监督模型迭代的研究者和开发者。
自改善的大语言模型（LLM）通过自身生成的合成数据进行微调，有望在减少人工标注的情况下提升模型能力。现有方法常依赖种子数据或第三方模型提供外部监督信号。本文提出Crescent框架，完全自主生成高质量问答数据：首先用诱饵提示引导模型生成原始问题，再通过基于拒绝采样的自去重机制增强多样性，最后将问题输入模型并采用多数投票方式收集答案。实验表明，Crescent在无需任何外部监督信号的情况下，实现了真正的自改善；具体而言，其生成的问答对不仅能有效提升模型的数学推理能力，同时保持其通用性能（尤其在零样本场景下）；此外，在知识蒸馏到较弱模型方面，优于基于种子数据增强的现有方法。
原文摘要 · Abstract (English)
Self-improving large language models (LLMs) -- i.e., to improve the performance of an LLM by fine-tuning it with synthetic data generated by itself -- is a promising way to advance the capabilities of LLMs while avoiding extensive supervision. Existing approaches to self-improvement often rely on external supervision signals in the form of seed data and/or assistance from third-party models. This paper presents Crescent -- a simple yet effective framework for generating high-quality synthetic question-answer data in a fully autonomous manner. Crescent first elicits the LLM to generate raw questions via a bait prompt, then diversifies these questions leveraging a rejection sampling-based self-deduplication, and finally feeds the questions to the LLM and collects the corresponding answers by means of majority voting. We show that Crescent sheds light on the potential of true self-improvement with zero external supervision signals for math reasoning; in particular, Crescent-generated question-answer pairs suffice to (i) improve the reasoning capabilities of an LLM while preserving its general performance (especially in the 0-shot setting); and (ii) distil LLM knowledge to weaker models more effectively than existing methods based on seed-dataset augmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。