在真实实验系统中测试因果表示学习方法，发现现有方法均失效。
Sanity Checking Causal Representation Learning on a Simple Real-World System
- 用可控光学实验构建真实数据集，提供已知因果因子作为真值
- 所有测试方法均无法恢复真实因果因子，即使在简化合成数据上也失败
- 揭示现有方法对混合函数假设过于敏感，与真实数据不匹配
我们在一个为评估因果表示学习（CRL）而专门设计的可控光学实验系统上测试了代表性CRL方法。该系统满足CRL的核心假设，且其输入（即潜在因果因子）已知，可作为真实基准。结果表明，所有被测方法均未能恢复出真实的因果因子。为进一步分析失败原因，我们对数据进行了消融：用更简单的合成过程替代真实生成机制。结果显示，多数方法在这一简化条件下仍表现不佳，暴露出可复现性问题。此外，我们发现某些方法性能严重依赖于对混合函数的常见假设，而这些假设在真实数据中并不成立。本工作凸显了当前先进CRL方法在理论承诺与实际应用之间的差距，旨在提供一个简单但真实的基准，以推动方法改进与验证。代码与数据集已在github.com/simonbing/CRLSanityCheck公开。
原文摘要 · Abstract (English)
We evaluate methods for causal representation learning (CRL) on a simple, real-world system where these methods are expected to work. The system consists of a controlled optical experiment specifically built for this purpose, which satisfies the core assumptions of CRL and where the underlying causal factors (the inputs to the experiment) are known, providing a ground truth. We select methods representative of different approaches to CRL and find that they all fail to recover the underlying causal factors. To understand the failure modes of the evaluated algorithms, we perform an ablation on the data by substituting the real data-generating process with a simpler synthetic equivalent. The results reveal a reproducibility problem, as most methods already fail on this synthetic ablation despite its simple data-generating process. Additionally, we observe that common assumptions on the mixing function are crucial for the performance of some of the methods but do not hold in the real data. Our efforts highlight the contrast between the theoretical promise of the state of the art and the challenges in its application. We hope the benchmark serves as a simple, real-world sanity check to further develop and validate methodology, bridging the gap towards CRL methods that work in practice. We make all code and datasets publicly available at github.com/simonbing/CRLSanityCheck
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。