arXiv:2608.10145cs.LG2026-08

复现发现评估协议影响结果,模型表现差异高达76%。

The Evaluation Protocol Determines the Result: An Independent Reproduction of LeWorldModel on TwoRoom

论文配图:The Evaluation Protocol Determines the Result: An Independent Reproduction of LeWorldModel on TwoRoom
图 1 · 摘自论文原文
  • 通过四个未公开的实验约定实现高精度复现
  • 独立测试达94.0%成功率,远超原论文84.0%
  • 评估方式与配置冲突,导致结果波动剧烈

LeWorldModel在TwoRoom环境上报告约87%目标达成率。我们用约25美元租用算力独立复现,所有评估在单台笔记本CPU上完成。在仓库设定的目标偏移下达到94.0%,而作者发布的检查点在相同测试集上仅得84.0%。直接复现了其表示结果(位置探测皮尔逊相关系数r=0.9988,报告值为0.996)。实现该结果需依赖四项未包含在发布配置中的约定:帧跳块内密集动作采集、程序设定的动作编码器宽度、ImageNet像素归一化、动作z-score标准化。仅按发布配置复现的模型预测无法收敛。评估协议本身存在矛盾:附录与代码库的配置给出不同目标偏移和步数预算;在作者权重上分别得到14.0%和84.0%的结果,仅配置值能复现报告数据。在50个相同测试轨迹上,仅改变目标构造方式,同一检查点得分从84.0%降至8.0%。两个结论具普遍性:单步预测准确率不能预测长程规划成功;三个检查点跨越七倍预测误差范围,单步准确率单调排序短程成功,却完全无法排序长程成功。批量归一化层使验证损失最高膨胀300倍,掩盖了全程平坦的训练损失。

原文摘要 · Abstract (English)

LeWorldModel trains a latent world model with a prediction loss and a single anti-collapse regulariser, and reports approximately 87% of goals reached on TwoRoom, its simplest diagnostic environment. We reproduce that result by independent reimplementation on roughly $25 of rented compute, with all evaluation on one laptop CPU. We reach 94.0% at the repository's evaluation goal offset, against 84.0% for the authors' own released checkpoint measured under our protocol on identical episodes, and we reproduce the reported representation result directly (position probe Pearson r = 0.9988 against a reported 0.996). Reaching that point required four conventions that determine the outcome and appear in no released configuration file: dense action gathering across a frameskip block, a programmatically-set action-encoder width, ImageNet pixel normalisation, and action z-scoring. A reproducer following the released configurations alone obtains a model whose predictor cannot converge. The evaluation protocol is itself contested by the released material. The paper's appendix and the repository's configuration specify different goal offsets and step budgets; on the authors' own weights these yield 14.0% and 84.0%, and only the configuration's values reproduce the reported figure. On fifty identical episodes, changing nothing but how the goal is constructed moves that checkpoint from 84.0% to 8.0%. Two findings generalise. One-step prediction accuracy does not predict long-horizon planning success: across three checkpoints spanning a sevenfold range in prediction error, including the authors' own, it orders short-horizon success monotonically and fails to order long-horizon success at all. And a batch normalisation layer inflated our reported validation loss by up to a factor of 300, concealing a training loss that was flat throughout.

复现研究评估协议世界模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。