arXiv:2512.20403cs.LG2025-12

用中间模型桥接大模型与小模型,低成本实现高性能推理。

BRIDGE: Budget-aware Reasoning via Intermediate Distillation with Guided Examples

  • 通过中间助手模型分阶段蒸馏知识,降低对昂贵API的依赖。
  • 在医疗、法律、金融任务中提升小模型性能28%-41%,减少10倍查询量。
  • 适合资源受限场景下部署高精度小模型的研究者与开发者。

将大型专有模型(如GPT-4)的知识蒸馏到参数少于10亿的小型可部署模型面临容量-预算困境:教师与学生间存在1000倍的容量差距,直接迁移效果差;而API成本又限制了数据收集。本文提出BRIDGE(基于引导示例的预算感知推理中间蒸馏),采用两阶段框架解决此问题。第一阶段,一个中等规模的教师助手(TA,约70亿参数)在严格限定的数据子集(如3%-5%)上学习,该子集通过零成本管道选取,仅依赖本地TA推理,兼顾信息熵难度与语义多样性。第二阶段,利用教师查询昂贵而TA推理免费的不对称性,由精炼后的TA为全量数据生成合成推理过程,训练小型学生模型。关键在于,在转移推理能力前,采用指令微调课程建立行为一致性。理论分析表明,当数据充足时,BRIDGE的泛化界优于直接蒸馏。在医学、法律和金融基准测试中,该方法实现28%-41%的性能提升,使学生模型与专有教师的差距缩小12%-16%,同时仅使用5%的教师查询量,远低于直接蒸馏基准所用100%预算。

原文摘要 · Abstract (English)

Distilling knowledge from large proprietary models (e.g., GPT-4) to tiny deployable models (less than 1B parameters) faces a critical capacity-budget trap: the 1000x capacity gap between teachers and students prevents effective direct transfer, while API costs prohibit extensive data collection. We introduce BRIDGE (Budget-Aware Reasoning via Intermediate Distillation), a two-phase framework that resolves these constraints through strategic intermediation and budget asymmetry. In Phase 1, a mid-sized Teacher Assistant (TA; e.g., about 7B) learns from the black-box teacher on a strictly limited subset of data (e.g., 3-5%), selected via a zero-API-cost pipeline that balances entropic difficulty and semantic diversity using only local TA inference. In Phase 2, we exploit this asymmetry-teacher queries are expensive, whereas TA inference is free to amplify supervision: the refined TA generates synthetic rationales for the full dataset to train the tiny student. Crucially, we apply an instruction-tuning curriculum to establish behavioral alignment in the tiny student before transferring reasoning. Our theoretical analysis shows that BRIDGE yields tighter generalization bounds than direct distillation when data is abundant. Experiments across medical, legal, and financial benchmarks demonstrate consistent improvements: BRIDGE delivers student performance gains of 28-41%, closing the capability gap with proprietary teachers by 12-16% while using 10x fewer teacher queries. Notably, BRIDGE defies the conventional cost-performance frontier, surpassing direct distillation baselines that use 100% of the budget while consuming only 5% of the resources.

知识蒸馏模型压缩推理优化预算约束

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。