用统计先行的门控架构,智能判断数据采购是否值得,降低错误决策风险。
SFGA: A Statistics-First Gating Architecture with Adjudicative Escalation for Trustworthy SFT Data Procurement
- 先基于多样性、效用、冗余三轴做低成本统计评估,达标才放行
- 准确率0.90,F1值0.83,每单位成本仅0.017美元,低于全量验证
- 引入正反方辩论机制,暴露模型偏见,适合高可靠性数据采购场景
采购监督微调(SFT)数据需在训练前决定候选语料是否值得购买。我们提出 extit{SFGA},一种以统计为核心的门控架构,将采购视为三个内在质量维度——多样性、效用与冗余——上的成本敏感路由问题。通过廉价的盲测量生成各维度的估计值及置信区间;仅当区间紧凑、样本量充足且三轴一致时,门控才通过决策,否则升级至由买方主张者与拒方主张者组成的仲裁辩论,由主审裁决。在包含12个数据集(3×2×2网格)的受控基准上,5个种子测试下,该门控实现0.90准确率和0.83 F1,每单位成本仅0.017美元,介于始终验证基线(0.75)与理想上限(0.98)之间,且低于始终升级(0.020美元)。进一步报告了辩论路径的诚实负面诊断:正方胜率0.80(p≈3×10⁻⁶),主张者互换后52%立场反转,揭示出朴素大模型裁判所隐藏的负面性与位置偏差。我们明确将注入控制键评估作为测量保真度与路由校准的合成基准,并将外部有效性列为未来工作。
原文摘要 · Abstract (English)
Procuring supervised fine-tuning (SFT) data forces a buyer to decide, before any downstream training, whether a candidate corpus is worth acquiring. We present \sys{}, a statistics-first gating architecture that treats procurement as a cost-aware routing problem over three intrinsic quality axes -- diversity, utility, and redundancy. Cheap blind measurements are summarised into per-axis estimates with confidence intervals; a gate accepts a decision only when intervals are tight, sample sizes are adequate, and the axes agree, otherwise it escalates the case to an adjudicative debate between a buy-advocate and a reject-advocate judge, resolved by a presiding verdict. On a controlled benchmark of 12 datasets ($2{\times}3{\times}2$ grid over the three axes) with 5 seeds, the gate reaches 0.90 accuracy and 0.83 $F_1$ at \$0.017 per unit, sitting between an always-verify baseline (0.75) and an oracle upper bound (0.98) while spending less than always-escalate (\$0.020). We further report honest negative diagnostics of the debate path: a con-side win rate of 0.80 ($p\approx3{\times}10^{-6}$) and a 52\% position-flip rate under advocate swapping expose negativity and positional biases that a naive LLM-judge would hide. We frame the injected-knob evaluation explicitly as a controlled synthetic benchmark for measurement fidelity and routing calibration, and delimit external validity as future work.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。