用可验证风险控制方法,平衡大模型的多个对立目标。
Conformal Arbitrage: Risk-Controlled Balancing of Competing Objectives in Language Models
- 通过数据驱动阈值,在主模型与保守守护者间动态切换。
- 在有限样本下保证错误率不超过用户设定上限。
- 无需改模型权重,适合实际部署中成本与安全权衡。
现代语言模型部署常需权衡多个相互冲突的目标,如有用性与无害性、成本与准确性、奖励与安全性。我们提出Conformal Arbitrage,一种后处理框架,通过学习数据驱动的阈值,在优化主要目标的主模型与符合安全护栏目标的保守守护者(可为另一模型或人工专家)之间进行调度。该阈值采用符合风险控制方法校准,提供有限样本、分布无关的保证:长期不可接受事件(如事实错误或安全违规)发生频率不会超过用户指定配额。由于该方法完全在API层运行,无需访问模型logits或更新权重,可与基于权重的对齐技术互补,并无缝集成至现有成本感知级联系统中。实验表明,Conformal Arbitrage能描绘出高效前沿,使用户在设定某一目标可接受性能水平的同时最大化另一目标的效用。结果表明,其准确率优于成本匹配的随机模型路由。这些特性使其成为可信且经济的大规模语言模型部署工具,适用于多种潜在冲突目标场景。
原文摘要 · Abstract (English)
Modern language model deployments must often balance competing objectives, for example, helpfulness versus harmlessness, cost versus accuracy, and reward versus safety. We introduce Conformal Arbitrage, a post hoc framework that learns a data driven threshold to mediate between a Primary model optimized for a primary objective and a more conservative Guardian which could be another model or a human domain expert aligned with a guardrail objective. The threshold is calibrated with conformal risk control, yielding finite sample, distribution free guarantees that the long run frequency of undesirable events, such as factual errors or safety violations, does not exceed a user specified quota. Because Conformal Arbitrage operates wholly at the API level, without requiring access to model logits or updating model weights, it complements weight based alignment techniques and integrates seamlessly with existing cost aware cascades. Empirically, Conformal Arbitrage traces an efficient frontier, allowing users to define an acceptable performance level for one objective while maximizing utility in another. We observe that our method outperforms, in terms of accuracy, cost matched random routing between models. These properties make Conformal Arbitrage a practical, theoretically grounded tool for trustworthy and economical deployment of large language models across a broad range of potentially competing objectives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。