用语义一致性判断模型输出可靠性，实现低成本高效大模型级生成
Semantic Agreement Enables Efficient Open-Ended LLM Cascades
- 通过多个模型输出的语义一致性判断是否可用小模型
- 在500M到70B参数模型间实现40%成本降低，延迟减少60%
- 无需修改模型、支持黑盒接口，适合实际部署
级联系统在可能时将计算请求分配给较小模型，仅在必要时调用大模型，是平衡大模型部署成本与质量的有前景方法。然而，在开放式文本生成中面临根本挑战：生成质量处于连续谱上，且存在多种有效答案，难以判断输出可靠性。为此，我们提出语义一致性的概念——即集成输出之间的意义共识，作为无需训练的可靠拒止信号。实验表明，当多个模型输出在语义上达成一致时，其共识比词级别置信度更能反映输出可靠性。在500M至70B参数模型上评估显示，语义级联系统在仅需目标模型40%成本的情况下达到或超越其性能，并将延迟降低高达60%。该方法不依赖模型内部结构，适用于黑盒API，且对模型更新保持鲁棒，可作为真实场景下大模型部署的实用基线。
原文摘要 · Abstract (English)
Cascade systems route computational requests to smaller models when possible and defer to larger models only when necessary, offering a promising approach to balance cost and quality in LLM deployment. However, they face a fundamental challenge in open-ended text generation: determining output reliability when generation quality lies on a continuous spectrum, often with multiple valid responses. To address this, we propose semantic agreement -- meaning-level consensus between ensemble outputs -- as a training-free signal for reliable deferral. We show that when diverse model outputs agree semantically, their consensus is a stronger reliability signal than token-level confidence. Evaluated from 500M to 70B-parameter models, we find that semantic cascades match or surpass target-model quality at 40% of the cost and reduce latency by up to 60%. Our method requires no model internals, works across black-box APIs, and remains robust to model updates, making it a practical baseline for real-world LLM deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。