用小模型在设备上实现高效文本提炼,不同教师带来不同能力提升
Different Teachers, Different Capabilities: Sub-1B On-Device Distillation for Structured Text Enrichment

- 用8B推理型教师蒸馏出0.6B小模型,显著降低推理时间
- 小模型摘要质量达到教师的58%,优于基线16.8分以上
- 推理型教师提升写作质量,管理型管道增强标签多样性
高负载结构化提取因大模型延迟过高而受限,将任务蒸馏至小型本地模型具有吸引力:以极低时间和成本获得接近的效果。本文评估了不同子任务下蒸馏的实际表现。每篇新闻文章被映射为一个包含简短摘要和五个类别标签的JSON对象。将8B推理型教师(deepseek-r1:8b)蒸馏至0.6B学生模型(Qwen3-0.6B;QLoRA,三组种子),并引入两种教师控制:同规模非推理教师与更大规模管理式流水线。由三位盲评员基于全文评分,对比完整文章与两个非蒸馏基线(少样本提示、约束解码)。学生模型每篇文章耗时约0.8秒,相较教师的39秒大幅降低;在摘要质量上恢复了基线与教师差距的58%,较约束解码提升+16.8分,较少样本提示提升+4.9分。同规模非推理教师训练的学生表现不及未调优基础模型,说明摘要提升源自教师的推理能力而非规模。能力分配因教师类型而异:推理型教师传递写作质量,管理流水线传递标签多样性;同规模指令型教师的学生在93项测试集中22篇短而信息稀疏的文章上更可靠(74对55忠实率),推理型学生则更易虚构。此差异为一致趋势而非整体显著效应,且子集较小,故仅作方向性报告。由于无单一模型通吃所有场景,最终交付物为面向本地增强的按任务路由图。
原文摘要 · Abstract (English)
High-volume structured extraction pays a large model's latency on every item, so distilling the task into a small on-device model is attractive: comparable output at a fraction of the time and cost. We measure what that distillation actually delivers, per sub-task. Each news article is mapped to one JSON object with a short summary and five categorical labels. We distill an 8B reasoning teacher (deepseek-r1:8b) into a 0.6B student (Qwen3-0.6B; QLoRA, three seeds), and add two teacher controls: a same-size non-reasoning teacher and a larger managed pipeline. A blinded, reference-free, three-judge panel scores every arm against the full article, alongside two non-distillation baselines, few-shot prompting and constrained decoding. The student runs at about 0.8 s per article against the teacher's 39 s, and recovers 58% of the base-to-teacher gap on summary quality, beating its primary baseline (constrained decoding) by +16.8 points and few-shot prompting by a secondary +4.9. A same-size non-reasoning teacher trains a student no better than the untuned base, so the summary gain follows from the teacher's reasoning nature rather than its scale. Capabilities then split by teacher: the reasoning teacher transfers writing quality and the managed pipeline transfers label diversity, while a same-size instruction teacher's students stay more grounded on the 22 short, thin-source articles in the 93-item test set (74 versus 55 faithful), where the reasoning-lineage student fabricates. That grounding difference is a consistent ordering rather than a significant aggregate effect, and the subgroup is small, so we report it as a direction. Because no single engine wins every field, the deliverable is a per-field routing map for on-device enrichment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。