在真实生产环境中,选择性使用外部经验可平衡质量与成本
External Experience Serving in Production LLM Systems: A Deployment-Oriented Study of Quality-Cost Trade-offs

- 通过检索筛选而非全局注入外部经验,实现更优的性能-成本权衡
- 当任务依赖具体案例时,选择性检索比全量注入效果更好
- 适合关注推理效率与成本控制的工业级大模型部署团队
生产级大模型系统积累了可复用的运行经验,但实际部署的关键问题不在于经验能否帮助,而在于不同服务策略如何在现实约束下权衡质量与在线成本。引入外部经验虽能提升任务质量,但也增加提示负担、延迟和服务压力。本文将外部经验服务视为一个面向部署的质量-成本权衡问题,在真实的生产内容审核场景中进行评估,并以工具使用和GPQA为对比任务,揭示不同输出-成本模式。比较了无经验基线、随机经验控制、全局提示注入与基于检索的选择性注入,分析任务质量与服务成本。结果表明,当经验具有案例依赖性时,选择性检索优于无条件全局注入;检索质量比单纯增大Top-K更重要;同一服务策略在短输出与解码密集型任务中呈现显著不同的成本收益特征。研究建议:外部经验应作为有选择的、成本敏感的服务决策,而非通用增强模块。总体而言,仅当服务接口与任务成本结构使质量提升足以覆盖在线开销时,外部经验才真正有效。
原文摘要 · Abstract (English)
Production LLM systems accumulate reusable operational experience, but the practical deployment issue is not merely whether such experience can help. It is how different serving strategies trade off quality against online cost under realistic constraints. Injecting external experience can improve task quality, yet it also increases prompt burden, latency, and serving pressure. We study \textit{external experience serving} as a deployment-oriented quality-cost trade-off problem. We evaluate this question in a real production moderation setting, with tool-use and GPQA as supporting contrast tasks that expose different output-cost regimes. We compare no-experience baselines, random experience controls, global prompt injection, and retrieval-based selective injection, and analyze both task quality and serving cost. The results show that, once experience becomes case-dependent, selective retrieval provides a stronger operating point than unconditional global injection. They further show that retrieval quality matters more than simply increasing Top-$K$, and that the same serving policy can exhibit substantially different cost-benefit profiles across short-output and decode-heavy regimes. These findings suggest that external experience is best treated as a selective, cost-aware serving decision rather than as a universal add-on. Overall, in the settings studied here, external experience pays off only when both the serving interface and the task-specific cost structure make its quality gains worth the online cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。