用可靠约束优化多模型协作，让大模型推理更省钱。
Conformal Constrained Policy Optimization for Cost-Effective LLM Agents
- 用可信预测约束成本，动态调度不同性价比的模型
- 在双基准测试中成本降低最高达30%，可靠性不变
- 适合需要低成本部署且对结果可靠性有要求的场景
尽管大语言模型在解决复杂人工智能问题上取得显著进展，但其计算和API成本持续攀升。本文提出一种新策略：通过智能编排多个具有不同成本/精度权衡的LLM模型，按序调用模型与工具，由一个协调模型决定执行路径，以在满足用户指定可靠性要求的前提下最小化成本；该约束通过置信预测（conformal prediction）形式化，提供可证明的保障。为此，我们提出符合约束的策略优化（CCPO），融合约束策略优化、离策略强化学习以及在线置信预测的最新进展。CCPO联合优化成本感知策略（评分函数）和自适应阈值。在两个多跳问答基准上，相比其他成本感知基线和基于LLM引导的方法，CCPO实现最高30%的成本降低，且不牺牲可靠性。本方法为部署高性价比且可靠的LLM代理提供了原理严谨且实用的框架。
原文摘要 · Abstract (English)
While large language models (LLMs) have recently made tremendous progress towards solving challenging AI problems, they have done so at increasingly steep computational and API costs. We propose a novel strategy where we combine multiple LLM models with varying cost/accuracy tradeoffs in an agentic manner, where models and tools are run in sequence as determined by an orchestration model to minimize cost subject to a user-specified level of reliability; this constraint is formalized using conformal prediction to provide guarantees. To solve this problem, we propose Conformal Constrained Policy Optimization (CCPO), a training paradigm that integrates constrained policy optimization with off-policy reinforcement learning and recent advances in online conformal prediction. CCPO jointly optimizes a cost-aware policy (score function) and an adaptive threshold. Across two multi-hop question answering benchmarks, CCPO achieves up to a 30% cost reduction compared to other cost-aware baselines and LLM-guided methods without compromising reliability. Our approach provides a principled and practical framework for deploying LLM agents that are significantly more cost-effective while maintaining reliability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。