多风险控制新方法,实现实时输出安全过滤。
MultiRisk: Multiple Risk Control via Iterative Score Thresholding
- 通过迭代阈值调整,动态管理多个风险约束。
- 在真实数据集上,各风险控制接近目标水平。
- 适合需要多维度安全保障的AI系统部署。
随着生成式AI在实际应用中的普及,对模型行为的多维度调控变得至关重要。本文聚焦于测试时过滤——一种轻量级的行为控制机制,通过将性能得分与估计阈值比较,在违反边界时修改输出。我们形式化了在用户定义优先级下施加多重风险约束的问题,并提出两种高效的动态规划算法。MULTIRISK-BASE 提供直接的有限样本阈值选择方法;MULTIRISK 则利用数据可交换性,保证所有风险的同步控制。在温和假设下,MULTIRISK 实现了几乎紧致的风险控制。分析采用复杂的迭代论证,通过引入多种对称化风险函数上界,并递归计数对称化风险函数在不同风险水平间的跃迁来下界风险。我们在 PKU-SafeRLHF 数据集上的三约束大语言模型对齐任务中评估该框架,目标是在满足多项安全约束的前提下最大化有用性,得分由大语言模型评判器和困惑度过滤器生成。实验表明,该算法能将近似达到目标水平地控制每个独立风险。
原文摘要 · Abstract (English)
As generative AI systems are increasingly deployed in real-world applications, regulating multiple dimensions of model behavior has become essential. We focus on test-time filtering: a lightweight mechanism for behavior control that compares performance scores to estimated thresholds, and modifies outputs when these bounds are violated. We formalize the problem of enforcing multiple risk constraints with user-defined priorities, and introduce two efficient dynamic programming algorithms that leverage this sequential structure. The first, MULTIRISK-BASE, provides a direct finite-sample procedure for selecting thresholds, while the second, MULTIRISK, leverages data exchangeability to guarantee simultaneous control of the risks. Under mild assumptions, we show that MULTIRISK achieves nearly tight control of all constraint risks. The analysis requires an intricate iterative argument, upper bounding the risks by introducing several forms of intermediate symmetrized risk functions, and carefully lower bounding the risks by recursively counting jumps in symmetrized risk functions between appropriate risk levels. We evaluate our framework on a three-constraint Large Language Model alignment task using the PKU-SafeRLHF dataset, where the goal is to maximize helpfulness subject to multiple safety constraints, and where scores are generated by a Large Language Model judge and a perplexity filter. Our experimental results show that our algorithm can control each individual risk at close to the target level.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。