拆解语音转指令数据流水线,发现质量提升不等于模型表现提升
A Factorial Ablation of a Speech-to-SFT Pipeline: Differential Effects on Data Quality and Downstream Transfer

- 构建可独立开关的两阶段语音转指令流水线,实现全因子实验设计
- 数据质量提升后,跨模型平均问答准确率未显著增长,仅领域对齐模型受益
- 揭示格式错配问题:生成解释性内容与事实回忆任务不匹配,适合领域适配研究者
工业界日益采用多阶段精炼的语音转监督微调(SFT)数据流水线,但尚未有公开的逐阶段消融研究,各阶段边际价值不明。本文设计一个生产就绪的语音转SFT流水线,其中语料重构(阶段0)和SFT数据质量优化(阶段2)可独立开关,形成2×2因子实验。在韩语医疗与金融会议录音上生成问答形式SFT数据,微调9个模型(5类大模型,参数量2.4B-70B),通过4名跨机构大模型评委、6位盲评专家及3个下游MCQA基准进行评估。核心发现:在固定标准SFT训练流程下,问答数据质量提升虽一致提高4名模型评委评分,但跨模型平均MCQA得分提升不显著;正向迁移集中于模型家族与领域匹配的组合。该差异模式与格式错配有关:阶段2使SFT数据更倾向解释性内容,而MCQA主要考察事实性记忆。六位人工评审均认定完整流水线数据质量更高,验证了大模型评委趋势。更换为Whisper-medium的语音识别引擎后,结果仍稳健。非幻觉审计显示,两个前沿大模型对约8%的问答表示未知。已发布样本、提示、代码及全部SFT检查点。
原文摘要 · Abstract (English)
Industry pipelines that turn speech into supervised fine-tuning (SFT) data via multi-stage refinement are increasingly adopted but, to our knowledge, have not been publicly ablated stage-by-stage, leaving each stage's marginal value unknown. We design a production-ready speech-to-SFT pipeline in which transcript refinement (Phase 0) and SFT data quality refinement (Phase 2) are independently toggleable, yielding a 2x2 factorial design. For each condition, we generate QA-form SFT data from Korean medical and finance conference recordings and fine-tune 9 models (5 LLM families, 2.4B-70B); we evaluate with four cross-provider LLM judges, a blind six-expert human evaluation, and 3 downstream MCQA benchmarks. Our central finding: under a fixed, standard SFT recipe, improvements in QA data quality do not transfer uniformly into downstream MCQA gains. 4-judge quality rises consistently, yet the cross-model mean MCQA gain is not significant; positive transfer concentrates on family-domain aligned pairs. This differential pattern is consistent with a format mismatch: Phase 2 shifts SFT-data composition toward explanatory items, while MCQA primarily probes factoid recall. All six human raters report higher full-pipeline quality, confirming the LLM-judge direction. An STT-engine swap to Whisper-medium confirms pipeline robustness. A non-hallucination audit shows the two frontier LLMs admit unknown on approximately 8% of QA on average; we release samples, prompts, code, and all SFT checkpoints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。