解决大模型批量生成时的重复问题,提升合成数据多样性。
Dynamic Context Evolution for Scalable Synthetic Data Generation
- 用模型自评过滤明显想法,保留新颖内容。
- 跨批次保持语义记忆,避免近似重复。
- 动态调整提示词,适合大规模数据生成场景。
大型语言模型在独立批量提示时会产生重复输出,这种现象称为跨批模式崩溃:当模型反复生成而无法访问之前结果时,输出多样性逐渐丧失。从业者长期依赖临时去重和种子轮换,但缺乏系统性框架。本文提出动态上下文演化(DCE),包含三种机制:(1) 口语化尾部采样(模型自评每个想法的显见程度,显见内容被丢弃),通过模型自我评估过滤高概率候选;(2) 语义记忆,维护持久嵌入索引,跨批次拒绝近似重复;(3) 自适应提示演化,基于记忆状态与轮换多样性策略重构每批生成提示。在三个领域(可持续包装概念、教育考试题、创意提示)及两个模型家族(gpt-5-mini 和 claude-haiku-4-5)上实验显示,组件消融测试中,每方法2-3个随机种子下,DCE实现0.0 ± 0.0%崩溃率,对比朴素提示的5.6 ± 2.0%;每种子生成17-18个HDBSCAN聚类,远超朴素方法的2-17个,表明概念结构更稳定。结果经all-MiniLM-L6-v2独立嵌入模型验证,且在VTS阈值tau和去重阈值delta敏感性测试中保持稳健。去重与提示演化单独无效,但联合有效,成本约0.50美元/1000候选,仅需标准API调用,无需微调或定制架构。
原文摘要 · Abstract (English)
Large language models produce repetitive output when prompted independently across many batches, a phenomenon we term cross-batch mode collapse: the progressive loss of output diversity when a language model is prompted repeatedly without access to its prior generations. Practitioners have long mitigated this with ad hoc deduplication and seed rotation, but no principled framework exists. We introduce Dynamic Context Evolution (DCE), comprising three mechanisms: (1) verbalized tail sampling (the model labels each idea with a guess about how obvious it is, and obvious ideas are discarded), which filters high-probability candidates via model self-assessment; (2) semantic memory, which maintains a persistent embedding index to reject near-duplicates across batches; and (3) adaptive prompt evolution, which reconstructs the generation prompt each batch using memory state and rotating diversity strategies. In experiments across three domains (sustainable packaging concepts, educational exam questions, and creative writing prompts) and two model families (gpt-5-mini and claude-haiku-4-5), a component ablation across 2-3 random seeds per method shows that DCE achieves 0.0 +/- 0.0% collapse versus 5.6 +/- 2.0% for naive prompting, while producing 17-18 HDBSCAN clusters per seed versus naive's volatile 2-17, indicating reliably richer conceptual structure. These results are validated with an independent embedding model (all-MiniLM-L6-v2) and hold across sensitivity sweeps of the VTS threshold tau and dedup threshold delta. Deduplication and prompt evolution are individually insufficient but jointly effective, at approximately $0.50 per 1,000 candidates using only standard API calls, with no fine-tuning or custom architectures required.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。