用合成数据测试大模型是否真懂物理规律，发现它常只记技巧不学原理。
What Has a Foundation Model Found? Using Inductive Bias to Probe for World Models
- 用假设的世界模型生成合成数据，检验大模型的归纳偏好是否匹配
- 训练中表现优异的模型在新任务上仍无法应用牛顿力学，说明未掌握底层规律
- 模型更依赖特定任务的表面规律，不适合需要泛化理解的科学建模场景
基础模型的核心假设是序列预测能揭示深层领域知识，如同开普勒预测行星运动最终引出牛顿力学。然而，评估这些模型是否真正捕捉到深层结构仍是难题。我们提出一种评估方法：通过从假定的世界模型生成合成数据，考察基础模型在适应新任务时的归纳偏好是否与该世界模型一致。这种方法被称为归纳偏好探针。在多个领域中，我们发现基础模型虽能在训练任务上表现良好，但在适配新任务时却未能发展出与底层世界模型一致的归纳偏好。尤其在轨道轨迹训练的模型，在面对新物理任务时始终无法应用牛顿力学。进一步分析表明，这些模型更倾向于形成任务特定的启发式规则，难以实现泛化。
原文摘要 · Abstract (English)
Foundation models are premised on the idea that sequence prediction can uncover deeper domain understanding, much like how Kepler's predictions of planetary motion later led to the discovery of Newtonian mechanics. However, evaluating whether these models truly capture deeper structure remains a challenge. We develop a technique for evaluating foundation models that examines how they adapt to synthetic datasets generated from some postulated world model. Our technique measures whether the foundation model's inductive bias aligns with the world model, and so we refer to it as an inductive bias probe. Across multiple domains, we find that foundation models can excel at their training tasks yet fail to develop inductive biases towards the underlying world model when adapted to new tasks. We particularly find that foundation models trained on orbital trajectories consistently fail to apply Newtonian mechanics when adapted to new physics tasks. Further analysis reveals that these models behave as if they develop task-specific heuristics that fail to generalize.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。