用小模型检测奖励欺骗，成本低且效果优于大模型。
Cheap Reward Hacking Detection
- 用小型Transformer编码器将轨迹映射到单位球面，距离近似奖励与元数据的L1差异。
- 在清洗后测试集上，检测准确率AUC达0.9467，5%假阳性率下真阳性率达0.8296。
- 不依赖自然语言推理，仅用轨迹信号即可高效识别奖励作弊，适合低成本部署。
一个小型Transformer编码器被训练，将终端-扳手（Terminal-Wrench）轨迹映射到单位球面，其中嵌入距离近似奖励信号与元数据信号之间的$L_1$距离。在该嵌入上添加线性探测器，在清洗后的测试集上实现AUC 0.9467和TPR@5%FPR 0.8296，匹配了经过清理的TW LLM作为裁判的AUC（0.9510），并在此信息条件下超越其TPR@5%FPR（0.7130 vs 0.8296），同时每轨迹成本降低约四个数量级。该编码器并非纯行为读取器：在探测时去除自然语言推理输入，使AUC下降至0.6213。
原文摘要 · Abstract (English)
A small transformer encoder is trained to map Terminal-Wrench trajectories onto a unit sphere where embedding distance approximates the $L_1$ distance between reward and metadata signals. A linear probe on top of that embedding detects reward hacking on the cleaned test split with AUC $0.9467$ and TPR@5%FPR $0.8296$, matching the TW sanitized LLM-as-judge AUC ($0.9510$ on the cleaned split) and exceeding its TPR@5%FPR ($0.7130$ vs $0.8296$) on the same information condition, at roughly four orders of magnitude lower per-trajectory cost. The encoder is not a pure behavior reader: stripping natural-language reasoning from its input at probe time drops AUC to $0.6213$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。