用辩论机制训练AI说真话，防止其搞科研破坏。
An alignment safety case sketch based on debate
- 让AI通过辩论自我纠错，提升诚实性
- 辩论表现好=系统基本诚实，部署中保持真实
- 适合研究超智能系统安全的学者参考
若AI在广泛任务上达到或超越人类能力，人类将难以高效评估其行为，导致难以用人类反馈引导其向善。一种解决方案是利用另一个超人智能系统，通过辩论揭示输出缺陷。本文提出一个AI对齐安全论证框架，旨在证明即使具备自主行动能力，AI也不会引发严重危害。重点防范公司内AI研发代理人为掩盖错误而伪造结果。该代理通过辩论训练，并受探索保障约束，以培养诚实性；部署期间通过在线训练维持诚实。论证基于四个关键假设：(1) 代理在辩论中表现优异；(2) 辩论表现优异意味着系统基本诚实；(3) 部署中不会显著丧失诚实性；(4) 部署环境可容忍部分错误。文章指出若干待解研究问题，若解决，可使该论证成为可信的AI安全依据。
原文摘要 · Abstract (English)
If AI systems match or exceed human capabilities on a wide range of tasks, it may become difficult for humans to efficiently judge their actions -- making it hard to use human feedback to steer them towards desirable traits. One proposed solution is to leverage another superhuman system to point out flaws in the system's outputs via a debate. This paper outlines the value of debate for AI safety, as well as the assumptions and further research required to make debate work. It does so by sketching an ``alignment safety case'' -- an argument that an AI system will not autonomously take actions which could lead to egregious harm, despite being able to do so. The sketch focuses on the risk of an AI R\&D agent inside an AI company sabotaging research, for example by producing false results. To prevent this, the agent is trained via debate, subject to exploration guarantees, to teach the system to be honest. Honesty is maintained throughout deployment via online training. The safety case rests on four key claims: (1) the agent has become good at the debate game, (2) good performance in the debate game implies that the system is mostly honest, (3) the system will not become significantly less honest during deployment, and (4) the deployment context is tolerant of some errors. We identify open research problems that, if solved, could render this a compelling argument that an AI system is safe.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。