用游戏环境检验大模型诚实性评估的可靠性,发现评估工具本身会影响结果。
Instrument Effects in Language-Model Honesty Evaluation: An Auditable Single-System Demonstration
- 构建文本冒险世界,由引擎判断任务是否可完成,模型仅作判断
- 同一条件下,评估规则变化使强结论从38/40降至7/40,不完整判定达28/40
- 揭示评估机制偏差,适合关注评估可信性的研究者阅读
语言模型诚实性评估将模型输出视为其真实认知证据。本文转而检验评估工具本身。构建了一个文本冒险世界,游戏引擎知晓任务是否可完成,语言模型在预算限制下需判断任务完成、不可达或未定;引擎对每次判断评分。决策规则在结果前记录,运行记录绑定执行过程,预注册强度因系列而异并公开。固定玩家时,评估工具选择显著改变行为表现。在四个字节完全相同的锚点上,将二类判断扩展为三类后,强结论从38/40降至7/40,新“未完成”判断占28/40;跨系列中,158个有效游戏中有93个以未完成结束。仅一句话披露成功条件,即令相同实例的错误判断从18/59降至0/58,因决策点减少且更清晰。同一配置重复运行在4个实例中有3个呈现非稳定判断分布,单次运行误将样本当作倾向。一个正式预注册的叙事风格梯度被证伪;两个事后生成的模式仍存:叙事风格存在使强结论数量翻倍,预算设置影响大于叙事内容(.383米灯 vs .150灯笼)。叙述者将丰富预算压缩向稀缺标记,但注册的中介测试结果为零。本文提出四步完整性验证协议用于评估工具。
原文摘要 · Abstract (English)
Evaluations of language-model honesty read the model's verdicts as evidence about the model. We test the instrument instead. We built a text-adventure world where the game engine, not any model, knows whether the quest can be completed. A language model plays under a budget and must eventually declare its quest complete, unreachable, or not yet decidable; the engine scores every verdict. Decision rules were recorded before results were read, and run artifacts bind the revisions they executed; the strength of preregistration varies by series and is disclosed. With the player held fixed, instrument choices substantially changed measured behavior. On four byte-identical anchors, expanding a two-verdict grammar to three verdicts moved strong claims from 38/40 to 7/40, while the new incomplete verdict took 28/40 outcomes; across series 2, 93/158 valid games ended incomplete. One sentence disclosing the success criterion took matched-instance false verdicts from 18/59 to 0/58, through fewer decision points and cleaner decisions. Repeated runs of one fixed configuration produced non-stable verdict distributions on 3 of 4 instances: single runs report samples as dispositions. A formally preregistered narrative-register gradient was falsified; two post-hoc, hypothesis-generating patterns remain: register presence roughly doubled strong claims, and budget rendering moved verdicts more than register content (.383 meter vs .150 lantern). The narrator compressed abundant budgets toward scarcity landmarks, yet the registered mediation test returned a null. We propose a four-check integrity protocol for eval instruments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。