通过中途淘汰低质生成,减少大模型合成数据的浪费。
Know When To Fold 'Em: Token-Efficient LLM Synthetic Data Generation via Multi-Stage In-Flight Rejection

- 在生成过程中分阶段检测并提前终止低质量文本
- 减少11%到78.2%的令牌消耗,准确率不降反升
- 无需训练或改架构,适合所有指令微调模型使用
尽管大语言模型(LLMs)用于合成数据在后训练流程中广泛应用,但现有方法通常在生成完整输出后再进行质量过滤,导致大量令牌被浪费在最终被丢弃的样本上。为解决此问题,我们提出多阶段中途淘汰(MSIFR),一种轻量、无需训练的框架,可在生成过程中的中间检查点识别并终止低质量轨迹,避免其完成生成。MSIFR将生成过程分解为连续阶段,利用快速规则式验证器检测算术错误、幻觉模式和格式违规,实现对故障样本的早期淘汰。我们将中途淘汰建模为序列决策过程,证明任何非平凡的丢弃策略都能降低预期令牌消耗,且淘汰越早,节节省越多。进一步证明条件效用估计构成鞅,确保早期淘汰不会偏倚保留样本的期望效用。在五个指令微调模型和七个推理基准上,MSIFR作为独立方法可减少11%-77%的令牌消耗,与早退机制结合时最高达78.2%,同时保持或提升评估准确率。结果表明,MSIFR为提升基于LLM的合成数据生成效率提供了实用方案,无需额外训练或结构改动。
原文摘要 · Abstract (English)
While synthetic data generation with large language models (LLMs) is widely used in post-training pipelines, existing approaches typically generate full outputs before applying quality filters, leading to substantial token waste on samples that are ultimately discarded. To address this, we propose Multi-Stage In-Flight Rejection (MSIFR), a lightweight, training-free framework that detects and terminates low-quality generation trajectories at intermediate checkpoints before they reach full completion. MSIFR decomposes the generation process into sequential stages and applies fast rule-based validators to identify arithmetic inconsistencies, hallucination patterns, and formatting violations, enabling early rejection of faulty samples. We formalize in-flight rejection as a sequential decision process and show that any non-trivial discard policy reduces expected token consumption, with stage-wise savings increasing when rejection occurs earlier in the generation pipeline. We further demonstrate that conditional utility estimates form a martingale, ensuring that early, in-flight rejection does not bias the expected utility of retained samples. Across five instruction-tuned models and seven reasoning benchmarks, MSIFR reduces token consumption by 11%-77% as a standalone method, and up to 78.2% when combined with early-exit methods, while preserving or improving evaluation accuracy. These results confirm that MSIFR provides a practical mechanism for improving the efficiency of LLM-based synthetic data generation without additional training or architectural changes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。