AI科研代理失控，需重构可验证科学体系
The Age of AI Agents Demands A New Scientific Paradigm To Sustain Trustworthy Science
- 要求科研流程公开可追踪，杜绝黑箱操作
- 必须实现大规模可验证，应对人类监督能力不足
- 适合关注AI可信科研的学者与政策制定者
AI系统正成为能自主提出假说、设计实验并产出发现的研究代理，其规模已超出人类监管能力。当前机器学习领域论文投稿量激增，科学产出与可验证性之间的差距已持续扩大，而人工智能代理因人机不对称性使问题恶化数倍。我们主张科学界必须升级验证基础设施，如历史上引入同行评审一般。但以往机制依赖可问责的人类参与者，而AI代理打破了这一前提。因此，我们提出新验证体系标准：默认可观测的工作流、可扩展的验证机制、清晰的责任归属。若不进行适应性变革，机器学习及其他使用代理的科学领域将面临严重风险：无人能验证的实验结果、为指标优化而牺牲理解、责任空白导致科学信任崩塌。
原文摘要 · Abstract (English)
AI systems are becoming autonomous research agents that generate hypotheses, design experiments, and produce discoveries at scales beyond human oversight. As seen by increased submissions to ML venues, the verification gap between scientific output and our ability to check it is already widening, and autonomous agents make it worse by magnitudes given human-agent asymmetry. We argue that science must evolve its verification infrastructure, as it has before with peer review. However, while historical adaptations assumed human contributors who could be questioned and sanctioned, AI agents break this assumption. We propose criteria for an adapted verification infrastructure that emphasizes observable-by-default workflows, scalable verification, and clear attribution. We argue that without adaptation, ML and any scientific domain using agents face dangerous failures: experimental results that no person can verify, optimization for metrics over understanding, and accountability vacuums that erode scientific trust.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。