小模型先答，信心不足再用大模型，省钱又准
Confidence-Calibrated Small-Large Language Model Collaboration for Cost-Efficient Reasoning
- 小模型先试答，信心低则转给大模型处理
- 成本降16.8%~21.5%，准确率仅降2%
- 用强化学习提升小模型信心判断能力
大语言模型（LLM）推理能力优于小语言模型（SLM），但成本更高。我们提出COREA系统，将SLM与LLM级联，实现复杂推理任务中精度与成本的平衡。COREA首先由SLM尝试回答问题，输出答案和口语化置信度评分；当置信度低于预设阈值时，问题转交至LLM进行更精确求解。我们引入基于强化学习的训练算法，通过额外的置信度校准奖励项，使SLM的置信度预测更准确。大量实验表明，该方法在多个数据集和模型架构上同时提升了SLM的推理能力和置信度校准效果。相比单独使用LLM，COREA在跨域数学和非数学数据集上分别降低21.5%和16.8%的成本，准确率下降不超过2%的绝对值。
原文摘要 · Abstract (English)
Large language models (LLMs) demonstrate superior reasoning capabilities compared to small language models (SLMs), but incur substantially higher costs. We propose COllaborative REAsoner (COREA), a system that cascades an SLM with an LLM to achieve a balance between accuracy and cost in complex reasoning tasks. COREA first attempts to answer questions using the SLM, which outputs both an answer and a verbalized confidence score. Questions with confidence below a predefined threshold are deferred to the LLM for more accurate resolution. We introduce a reinforcement learning-based training algorithm that aligns the SLM's confidence through an additional confidence calibration reward. Extensive experiments demonstrate that our method jointly improves the SLM's reasoning ability and confidence calibration across diverse datasets and model backbones. Compared to using the LLM alone, COREA reduces cost by 21.5% and 16.8% on out-of-domain math and non-math datasets, respectively, with only an absolute pass@1 drop within 2%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。