arXiv:2601.20103cs.SEcs.AI2026-01被引 13

提出代码环境奖励欺骗检测新基准,发现对比学习能显著提升检测效果。

Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysis

  • 构建517条轨迹的对比式检测基准TRACE,涵盖54类奖励欺骗。
  • 在对比设置下模型检测率最高达63%,比孤立分类提升18个百分点。
  • 揭示语义级欺骗更难检测,适合评估大模型安全性与评测工具开发者。

强化学习在代码生成中的进展使得鲁棒性环境至关重要,以防止奖励欺骗。随着大语言模型越来越多地作为代码生成强化学习中的评估器,其检测奖励欺骗的能力仍缺乏研究。本文提出一个覆盖54个类别的奖励欺骗新分类体系,并引入TRACE(Testing Reward Anomalies in Code Environments)——一个经合成构造与人工验证的基准,包含517条测试轨迹。不同于以往在孤立分类场景中评估奖励欺骗检测的方法,本文采用更具现实意义的对比异常检测设置。实验表明,在对比设置下模型对奖励欺骗的识别能力显著优于孤立分类,其中GPT-5.2在最高推理模式下检测率达63%,较孤立设置的45%有明显提升。进一步分析显示,当前最先进模型在语义上下文化的奖励欺骗面前表现远逊于语法上下文化的情况。此外,定性分析与消融实验表明,良性与恶意轨迹比例及分析聚类规模对检测性能有显著影响。本文已开源基准与评估工具,以促进社区持续扩展和评测。

原文摘要 · Abstract (English)

Recent advances in reinforcement learning for code generation have made robust environments essential to prevent reward hacking. As LLMs increasingly serve as evaluators in code-based RL, their ability to detect reward hacking remains understudied. In this paper, we propose a novel taxonomy of reward exploits spanning across 54 categories and introduce TRACE (Testing Reward Anomalies in Code Environments), a synthetically curated and human-verified benchmark containing 517 testing trajectories. Unlike prior work that evaluates reward hack detection in isolated classification scenarios, we contrast these evaluations with a more realistic, contrastive anomaly detection setup on TRACE. Our experiments reveal that models capture reward hacks more effectively in contrastive settings than in isolated classification settings, with GPT-5.2 with highest reasoning mode achieving the best detection rate at 63%, up from 45% in isolated settings on TRACE. Building on this insight, we demonstrate that state-of-the-art models struggle significantly more with semantically contextualized reward hacks compared to syntactically contextualized ones. We further conduct qualitative analyses of model behaviors, as well as ablation studies showing that the ratio of benign to hacked trajectories and analysis cluster sizes substantially impact detection performance. We release the benchmark and evaluation harness to enable the community to expand TRACE and evaluate their models.

奖励欺骗代码生成评测基准大模型安全

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。