揭示任务框架如何悄悄改变大模型的决策信念,影响长期表现。
Measuring Harness-Induced Belief Divergence in Multi-Step LLM Agents

- 设计信念回溯诊断法,追踪多步决策中的信心变化
- 发现不同框架下成功率不变,但决策信念显著偏离
- 适合评估智能体时关注隐藏偏见的研究者和工程师
当前软件代理评测通常只记录任务是否完成,却忽视了任务框架对代理感知与行为的深层影响:包括可见信息、可执行动作、失败修复方式、状态验证策略和证据记录机制。我们发现,在固定任务、环境与基础大模型的前提下,仅改变框架设计即可引发代理在多步推理中信念的根本性偏差。为此提出信念回溯诊断方法,系统提取在不同框架下关于进展、风险、可恢复性、约束、故障模式、不确定性、未来成功概率、修复成本及下一步行动等维度的结构化K步轨迹。定义跨框架信念差异,并分解为即时界面切换带来的到达项与随时间演进的生长项。在受控编码任务与公开基准压力测试中,受限动作、压缩修复、选择性验证及成本敏感的证据修剪等策略常能维持最终成功,却显著改变了驱动后续决策的内部信念。进一步提出无需训练的BIWM协议,通过归一化观测、记录被截断分支、扩展修复路径、标记验证掩码、在影子中执行高风险分支、对齐不同框架下的信念轨迹,实现更透明的评估。结果表明,框架设计应被视为代理评估中的关键实验变量,而非技术细节。代码已开源。
原文摘要 · Abstract (English)
Software-agent benchmarks usually report whether an agent solves a task, but the agent reaches that outcome through a harness that controls what it sees, which actions it can take, which failures are repaired, which states are verified, and which evidence is logged. We show that this harness can change the agent's multi-step beliefs even when the task, environment, and base LLM are fixed. We introduce a belief-rollout diagnostic that elicits structured K-step trajectories over progress, risk, recoverability, constraints, failure mode, uncertainty, future success, repair cost, and next action under alternative harnesses. We define a cross-harness belief divergence and decompose it into an arrival term for immediate interface shifts and a growth term for horizon-dependent belief changes. On controlled coding tasks and public-benchmark stress tests, blocked actions, compressed repairs, selective verification, and cost-aware evidence pruning often preserve terminal success while changing the beliefs that drive later decisions. We further introduce BIWM, a no-training protocol that canonicalizes observations, logs censored branches, expands repair traces, records verification masks, executes risky branches in shadow, and aligns belief trajectories across harness views. The results suggest that harness design is an experimental variable in agent evaluation, not an implementation detail. Our code is available at https://github.com/Hik289/Harness-induce-bias.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。