用推理时计算和部署模拟框架提升安全评估的真实性。
Improving Evaluation Realism with Inference-Time Compute and Deployment Scaffolds
- 通过多次生成并优化候选动作，使模拟更接近真实部署。
- 在多个模型上验证，双技术组合效果优于单一方法。
- 适合关注评估可信度的AI安全研究者使用。
对齐评估的核心障碍是评估感知：强大模型能识别自己处于测试而非部署状态，从而削弱安全评估结论的有效性。本文提出两种技术，使模拟对齐评估更难与真实部署区分。第一种为批判精炼，即在每次模拟动作时增加推理阶段的计算量：模拟器生成多个候选动作，利用目标模型实例提供的反馈进行优化，最终选择最贴近真实部署的动作继续评估。第二种为DISH（仿部署SWE-Agent测试框架），通过构建代理框架来缩小编码场景中模拟与真实部署环境的差距。我们在多个目标模型上测试了这些技术，发现二者可协同增效，联合应用带来的真实性提升显著超过单独使用。结果表明，自动化方法能有效提升对齐评估的真实感，且相比延长审计时间，更高效地利用额外计算资源。
原文摘要 · Abstract (English)
A core obstacle to alignment evaluation is evaluation awareness: capable models can tell when they are being tested rather than deployed, weakening the conclusions a safety evaluation can support. We present two techniques that make simulated alignment evaluations harder to distinguish from real deployments. Our first technique, critique refinement, spends additional inference-time compute on each simulator action: the simulator generates multiple candidate actions, refines them using feedback from an instance of the target model on how to make them more realistic, and continues the evaluation with the most deployment-like candidate. Our second technique, DISH (Deployment-Imitating SWE-Agent Harness), wraps the target in an agent harness, reducing the gap between simulated and real deployment environments in coding settings. We test the techniques on multiple target models and find that they compose: applying both yields larger realism gains than either alone. Our results show that automated approaches can improve the realism of alignment evaluations, and that these improvements use additional compute more effectively than making the audits longer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。