arXiv:2510.09330cs.LG2025-10

不重训不碰模型内部,用博弈论让大模型生成安全又有用的回复。

Safety Game: Inference-Time Alignment of Black-Box LLMs via Constrained Optimization

  • 将安全与有用性冲突建模为零和博弈,求解最小最大均衡策略。
  • 在推理时通过线性规划计算最优平衡点,实现黑箱模型安全对齐。
  • 无需访问模型参数,适合资源有限的第三方机构快速部署。

确保大语言模型(LLM)符合安全要求是人工智能部署的核心挑战。现有对齐方法主要依赖训练阶段的微调或基于人类反馈的强化学习,但成本高且缺乏灵活性,新需求出现时需重新训练。近期推理时对齐方法虽缓解部分问题,但仍需访问模型内部,不适用于无模型访问权限的第三方。本文提出一种无需重训、不依赖模型内部结构的黑箱对齐框架。以安全与信息量权衡为例,将该矛盾建模为双人零和博弈,其极小极大均衡捕捉了安全与有用性的最佳平衡。通过在线性规划求解器支持下,由LLM代理在推理时计算均衡策略。实验表明该方法可行,为小型组织及资源受限实体提供了可扩展、易获取的安全保障路径,适用于快速演进的大型语言模型生态系统。

原文摘要 · Abstract (English)

Ensuring that large language models (LLMs) comply with safety requirements is a central challenge in AI deployment. Existing alignment approaches primarily operate during training, such as through fine-tuning or reinforcement learning from human feedback, but these methods are costly and inflexible, requiring retraining whenever new requirements arise. Recent efforts toward inference-time alignment mitigate some of these limitations but still assume access to model internals, which is impractical, and not suitable for third party stakeholders who do not have access to the models. In this work, we propose a model-independent, black-box framework for safety alignment that does not require retraining or access to the underlying LLM architecture. As a proof of concept, we address the problem of trading off between generating safe but uninformative answers versus helpful yet potentially risky ones. We formulate this dilemma as a two-player zero-sum game whose minimax equilibrium captures the optimal balance between safety and helpfulness. LLM agents operationalize this framework by leveraging a linear programming solver at inference time to compute equilibrium strategies. Our results demonstrate the feasibility of black-box safety alignment, offering a scalable and accessible pathway for stakeholders, including smaller organizations and entities in resource-constrained settings, to enforce safety across rapidly evolving LLM ecosystems.

安全对齐黑箱模型博弈论推理时

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。