让AI研究像模糊测试一样,靠中间反馈探索而非盲目试错。
The Greatness of Science Cannot Be Planned: Agentic Auto-Research is Fuzz Testing

- 用可追踪的中间进展信号替代最终评分,指导研究方向。
- 在模拟物理环境中发现隐藏规律,优化基线因过拟合失败。
- 适合关注可信科学发现与抗过拟合的AI研究者。
代理式自动研究正兴起,但多数系统将科学发现视为以最终基准为目标的优化,仅奖励稀疏的最终判断,忽视前期探索。当代理只优化最终得分时,会过拟合测试条件并盲目采样而非有效搜索。在明确的研究问题下,研究代理与软件分析中的灰盒模糊测试面临相同的稀疏反馈。模糊测试很少直接发现漏洞,但每次执行都能通过覆盖率暴露部分进展。模糊测试利用这种密集信号进行输入变异和资源分配,而非仅对完成实验进行排序。自动研究也需要这两项能力:第一,每个实验必须在最终科学验证前提供廉价且密集的认知进展信号;第二,该信号应决定下一步干预,使代理能搜索而非重复采样。由于进展信号提供引导而非最终结论,最终验证仍需使用免于自适应重用的证据来评估主张。我们提出控制性实验,检验候选信号是否预测已验证进展、反馈驱动搜索是否单位成本产生更多验证发现,以及受保护验证是否减少虚假发现。在模拟物理环境中,追踪中间认知进展的AI研究代理发现了隐藏的物理定律。以优化为导向的基线因反复采样和过拟合现有数据而失败。反馈架构而非生成能力,是自动研究的核心瓶颈。
原文摘要 · Abstract (English)
Agentic auto-research is emerging, but most systems treat scientific discovery as goal-oriented optimization against a final benchmark. This paradigm rewards a sparse final verdict and ignores the exploration that precedes it. When agents optimize only the final score, they overfit to the test conditions and sample blindly rather than search. Within a declared research problem, a research agent and a greybox fuzzer for software analysis face the same sparse feedback. A fuzzer rarely finds a bug directly, but coverage makes partial progress observable on every execution. Fuzzers use that dense signal to mutate inputs and allocate effort, rather than merely rank completed runs. Auto-research needs the same two capabilities. First, each experiment must expose a cheap, dense signal of epistemic progress before final scientific validation is available. Second, that signal must determine the next intervention so the agent searches rather than repeatedly samples. Because the progress signal provides guidance rather than a final verdict, final validation must still evaluate claims using evidence protected from adaptive reuse. We propose controlled tests to determine whether candidate signals predict validated progress, whether feedback-directed search yields more validated discoveries per unit cost than repeated sampling, and whether protected validation reduces false discoveries. In a simulated physics environment, an AI research agent that tracks its intermediate epistemic progress discovers a hidden physical law. Optimization-driven baselines fail because they repeatedly sample and overfit to their existing data instead of probing unfamiliar regimes. Feedback architecture, not generation capacity, is the central bottleneck in auto-research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。