用重组实体对生成更丰富的领域问答数据,提升小模型微调效果。
EmbGen: Teaching with Reassembled Corpora

- 将语料拆解为实体-描述对,基于嵌入相似性重新组合
- 在高异质数据上,2000万token预算下准确率提升88.9%
- 适合需要高质量合成数据的领域小模型适配场景
将小规模指令微调模型适应特定领域通常依赖于人工标注的指令-响应样本,成本高昂。利用教师大模型从领域语料生成合成数据可降低成本,但现有方法易产生同质化输出,且难以捕捉跨段落或跨文档依赖关系。我们提出EmbGen,一种合成数据生成流程:将语料分解为实体-描述对,基于嵌入相似性推断语义结构进行重组,并通过邻近、簇内与簇间采样结合簇专用系统提示生成问答对。在三个语义异质性不同的数据集上,固定500万与2000万令牌预算下,对比EntiGraph、InstructLab和Knowledge-Instruct。采用词面重叠度、大模型判别评分及二元准确率(结合事实准确性与完整性)评估。EmbGen在最异质数据集上分别较最强基线提升12.5%(500万)和88.9%(2000万),在其他低异质性数据集上表现仍具竞争力。
原文摘要 · Abstract (English)
Adapting small instruction-tuned models to specialized domains often relies on supervised fine-tuning (SFT) on curated instruction-response examples, which is expensive to collect at scale. Synthetic training examples generated by a teacher LLM from a domain corpus can reduce this cost, but existing pipelines can produce homogenized outputs and do not consistently capture cross-passage or cross-document dependencies. We introduce EmbGen, a synthetic data generation pipeline that decomposes a corpus into entity-description pairs, reassembles them using semantic structure inferred from embedding similarity, and then generates question-answer (QA) pairs via proximity, intra-cluster, and inter-cluster sampling with cluster-specialized system prompts. We evaluate EmbGen against EntiGraph, InstructLab and Knowledge-Instruct on three datasets of varied semantic heterogeneity, under fixed token budgets (5 and 20 million tokens). We use lexical overlap metrics, an LLM-as-a-judge rubric, and Binary Accuracy, a composed metric combining Factual Accuracy and Completeness for evaluation. EmbGen improves Binary Accuracy on the most heterogeneous dataset by 12.5% at 5M and 88.9% at 20M tokens budget, relative to the strongest baseline, while remaining competitive across other datasets with lower heterogeneity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。