用精心设计的提示词生成大量合成数据，提升大模型在专业领域的知识覆盖。
SPA: A Simple but Tough-to-Beat Baseline for Knowledge Injection
- 通过少量精心设计的提示词生成大规模合成数据，实现知识注入。
- 在多个任务上超越现有强基线方法，尤其在数据规模扩大时表现更优。
- 揭示了强化学习与多阶段提示的局限性，强调提示设计的重要性。
尽管大语言模型（LLMs）在海量数据上进行了预训练，其在专业、数据稀缺领域中的知识覆盖仍不完整，这促使研究者广泛探索用于知识注入的合成数据生成方法。本文提出SPA（Scaling Prompt-engineered Augmentation），一种简单但难以被超越的基线方法，利用少量精心设计的提示词生成大规模合成数据以实现知识注入。通过系统对比，我们发现SPA优于多个强基线方法。此外，我们识别出先前方法的两个关键局限：(1) 虽然基于强化学习的方法在小规模下可提升令牌效率，但在数据规模增大时会因多样性崩溃而出现收益递减；(2) 尽管多阶段提示可能优于简单增强方法，但其优势在经过充分提示调优后会消失。结果表明，对于知识注入任务，精心设计的提示词结合直接的大规模增强策略具有出人意料的有效性。我们希望SPA能成为该领域未来研究的有力基线。代码已公开于 https://github.com/Tangkexian/SPA。
原文摘要 · Abstract (English)
While large language models (LLMs) are pretrained on massive amounts of data, their knowledge coverage remains incomplete in specialized, data-scarce domains, motivating extensive efforts to study synthetic data generation for knowledge injection. We propose SPA (Scaling Prompt-engineered Augmentation), a simple but tough-to-beat baseline that uses a small set of carefully designed prompts to generate large-scale synthetic data for knowledge injection. Through systematic comparisons, we find that SPA outperforms several strong baselines. Furthermore, we identify two key limitations of prior approaches: (1) while RL-based methods may improve the token efficiency of LLM-based data augmentation at small scale, they suffer from diversity collapse as data scales, leading to diminishing returns; and (2) while multi-stage prompting may outperform simple augmentation methods, their advantages can disappear after careful prompt tuning. Our results suggest that, for knowledge injection, careful prompt design combined with straightforward large-scale augmentation can be surprisingly effective, and we hope SPA can serve as a strong baseline for future studies in this area. Our code is available at https://github.com/Tangkexian/SPA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。