让弱模型在线学习强模型的解题思路,提升效率与准确率。
From Deferral to Learning: Online In-Context Knowledge Distillation for LLM Cascades
- 强模型解决难题后生成通用策略存入动态库,供弱模型复用。
- 弱模型准确率最高提升33.06%,强模型调用减少48.05%。
- 无需微调即可实现知识迁移,适合各类大模型部署场景。
标准的LLM级联通过将复杂查询从弱模型转给强模型来提升效率,但这类系统通常是静态的:面对重复或语义相似的查询时,会反复调用昂贵的强模型,无法在推理过程中自适应。为此,我们提出Inter-Cascade——一种在线交互式框架,将强模型从临时助手转变为长期教师。当强模型处理完被延迟的查询后,会生成可泛化的、可复用的问题求解策略,并存入动态存储库,未来通过相似性匹配为弱模型提供上下文增强。这使弱模型能在不进行昂贵参数微调的情况下“边工作边学习”。理论上,该机制提升了弱模型的置信度校准能力。实验证明,Inter-Cascade在多个基准上优于传统级联,弱模型准确率最高提升33.06%,系统整体准确率提升6.35%,强模型调用减少48.05%,费用节省高达49.63%。该方法实现了大模型间的有效上下文知识迁移,是一种通用且可扩展的框架,适用于开源及API型大模型。
原文摘要 · Abstract (English)
Standard LLM cascades improve efficiency by deferring difficult queries from weak to strong models. However, these systems are typically static: when faced with repeated or semantically similar queries, they redundantly consult the expensive model, failing to adapt during inference. To address this, we propose Inter-Cascade, an online, interactive framework that transforms the strong model from a temporary helper into a long-term teacher. In our approach, when the strong model resolves a deferred query, it generates a generalized, reusable problem-solving strategy. These strategies are stored in a dynamic repository and retrieved via similarity matching to augment the weak model's context for future queries. This enables the weak model to learn on the job without expensive parameter fine-tuning. We theoretically show that this mechanism improves the weak model's confidence calibration. Empirically, Inter-Cascade outperforms standard cascades on multiple benchmarks, improving weak model and overall system accuracy by up to 33.06 percent and 6.35 percent, while reducing strong model calls by up to 48.05 percent and saving fee by up to 49.63 percent. Inter-Cascade demonstrates effective in-context knowledge transfer between LLMs and provides a general, scalable framework applicable to both open-source and API-based LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。