用跨供应商审计确保AI科研的客观性,记录可追溯可复现。
CrossAudit: A Git-Native, Cross-Vendor Audit Loop for Agentic Science
- 不同厂商模型交叉审核,规则由人工编写并版本化
- 审计过程全记录为Git提交,可回溯、引用、不可篡改
- 适合关注AI科研可信度与透明性的研究者
AI科学家不应自我评判。我们发现现有系统中,评审模型通常与生成模型同源或同厂商,而模型评估常偏袒自身生成结果。若同类模型存在共性盲点,评审者可能继承作者缺陷。审核记录常藏于平台日志,外部无法复现。为此提出CrossAudit协议:每轮工作由不同厂商的模型依据人工编写的规则书进行审计,报告、裁决、争议与裁定均以Git提交形式存档,监督历史可重读可引用;原始模型交互暂不纳入记录。预设检查在模型执行前运行。建议性判断不阻塞流程:模型仅能因引用规则阻塞,且不得豁免确定性失败。持续多次修订后仍被阻塞的事项将移交人工。协议定义八条不变式。实现基于GitHub Actions与数百行Python代码,并部署于计算化学流水线。开展种子缺陷测试(30个步骤,43个预设缺陷,每配置运行一次),对论文仓库自身进行跨厂商审计后揭示了盲点。采纳该审计结果并修正数据。测试表明两厂商对同一规则理解不同,但未证明孰优孰劣。最强证据是本论文自身经受的、已提交的、不受控制的跨厂商审计记录。
原文摘要 · Abstract (English)
An AI scientist should not grade its own homework. Yet in the systems we examined, the agent that reviews the work usually comes from the same model family as the agent that produced it, or at least from the same vendor. Model evaluators are known to favour their own generations. Whether models trained alike also share blind spots is a conjecture, not a settled finding, but if they do, the reviewer inherits the author's. The record of what was flagged and what was waved through often sits in platform logs that nobody outside can replay. We present CrossAudit, a protocol for supervising autonomous research pipelines. It rests on three commitments. Each increment of work is audited by an agent from a different vendor against a rulebook a human wrote and versioned. Reports, verdicts, disputes and rulings are git commits, so the supervision history can be re-read and cited; raw model exchanges are not yet part of that record. Scripted checks run before any model does. Advisory judgement never gates the pipeline: a model blocks only by citing a rule, and no model may waive a deterministic failure. Blockers that survive a bounded number of revision rounds go to a person. We state the protocol as eight invariants. We describe a reference implementation built from GitHub Actions and a few hundred lines of Python, and report a live deployment of a closely related variant in a computational-chemistry pipeline. We also ran a seeded-defect trial (30 increments, 43 seeded defects, one run per configuration). A cross-vendor audit of our own repository then voided its blinding. We adopt that audit's findings and report the corrected results. The trial shows that two vendors read the same rulebook differently. It does not show that either is better. The strongest evidence here is the committed, uncontrolled record of cross-vendor audits of this paper itself.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。