用人类经验约束解空间，提升长反馈周期任务的模型性能
The FIL Hypothesis: Inductive Biases Help with Kernel Engineering
- 引入反馈信息循环时长作为新尺度，提出人类知识可弥补数据驱动局限
- 在需数小时至数周验证的GPU编程任务中，带归纳偏置的方法优于纯数据驱动
- 适合物理世界、科学计算等长反馈周期场景，为未来AI提供新范式
《Bitter Lesson》认为，通用方法随计算与数据规模增长终将超越依赖人工知识的方法。我们重新审视这一原则，发现一个关键的新缩放维度：反馈信息循环（FIL），即系统生成预测后获得验证信号所需时间。历史上绝大多数人工智能成功案例均依赖即时反馈（如游戏或分类任务），但未来科学与物理世界的应用将天然面临从数小时到数周不等的长FIL。这带来了根本性缩放限制——纯数据驱动方法所需验证步骤难以实现。为此，我们提出一种与数据驱动方法正交的方法：基于人类启发的专家知识，利用归纳偏置约束解空间。通过在具有显著长FIL的实世界GPU编程任务上验证，证明引入归纳偏置可显著优于纯数据驱动方法。代码已开源：https://github.com/ai-nikolai/robust_kernelbench
原文摘要 · Abstract (English)
The Bitter Lesson, which posits that general-purpose methods that scale with computation and data ultimately outperform those with built-in human knowledge, has become a dominant paradigm in the era of Large Language Models. We revisit this principle by observing a new and critical scaling dimension: the duration of the Feedback Information Loop (FIL), the time required for a system to receive a verification signal after generating a prediction. Most historic successes in Artificial Intelligence (AI) have benefited from near instantaneous feedback (e.g., games or classification tasks), but we argue that future AI applications in science and the physical world will inherently involve FILs ranging from hours to weeks. This trend poses a fundamental scaling limit, as obtaining enough verification steps required by purely data-driven methods becomes practically impossible. Additionally, we propose a method that is orthogonal to purely data-driven approaches, based on human-inspired expert knowledge. The method relies on inductive biases and constraining the solution space. We provide an initial validation of the hypothesis and the method, by studying the real-world GPU programming task, a domain with non-trivial FIL, and demonstrate that incorporating inductive biases yields superior performance over data-driven approaches. The code is released under: https://github.com/ai-nikolai/robust_kernelbench
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。