测试大模型在已知漏洞文件中复现真实漏洞的能力,发现效果普遍不佳。
Benchmarking Mythos-Linked Bug Rediscovery

- 用同一组文件和工具,让三款模型重复尝试复现六个系统漏洞。
- 最佳模型仅成功复现5次,覆盖2个任务,共识别出3个核心漏洞。
- 模型常误判其他候选问题,错过官方补丁修复的关键变量。
Anthropic 2026年4月发布的Mythos资料将基准声称与OpenBSD、FreeBSD、Linux、FFmpeg及浏览器中的具体漏洞发现故事结合。本文对六个公开或高可信度的Mythos关联系统任务开展受控的目标文件复现实验。每个模型接收相同的待测文件、只读源码工具,每项任务重复三次,采用统一的手动目标匹配标准;提示词中不包含CVE编号、补丁哈希、公告文本、作者姓名、披露日期及答案根因语言。实验共产生54次计数尝试:三款模型、六项任务、三次重复,每模型18次尝试。GPT-5.5 xhigh实现5/18次目标复现,覆盖2/6任务;若将一次错误目标mpegts.c的发现单独计数,则共识别出3/6个不同核心漏洞。Claude Opus 4.7实现1/18次目标复现,覆盖1/6任务。Kimi K2实现0/18次目标复现。主要失败模式为过早锁定合理但错误的备选候选对象:模型常基于源码提出假设,却遗漏了公共Mythos补丁证据所修正的具体不变量。这些结果并未否定Anthropic未公开的工作流,但在这一有利的目标文件框架下表明,系统特定提示策略在54次尝试中仅获得六次目标匹配。
原文摘要 · Abstract (English)
Anthropic's April 2026 Mythos materials combine benchmark claims with concrete bug-finding stories across OpenBSD, FreeBSD, Linux, FFmpeg, and browsers. This paper reports a controlled target-file rediscovery experiment on six public or high-confidence Mythos-linked systems tasks. Each model receives the same target file or files, read-only source tools, three repeats per task, and one manual target-matching rubric; prompts omit CVE identifiers, patch hashes, advisory text, author names, disclosure dates, and answer key root cause language. The experiment contains 54 counted model-task attempts: three models, six tasks, and three repeats, giving 18 attempts per model. GPT-5.5 xhigh achieves 5/18 target rediscoveries, covering 2/6 tasks; counting one wrong-target mpegts.c finding separately gives 3/6 distinct core bugs. Claude Opus 4.7 achieves 1/18 target rediscoveries, covering 1/6 tasks. Kimi K2 records 0/18 target rediscoveries. The dominant failure mode is early commitment to plausible alternate candidates within the assigned file: models often submit source-grounded hypotheses while missing the specific invariant corrected by public Mythos patch evidence. These results do not refute Anthropic's undisclosed workflow, but show that under this favorable target-file scaffold, systems-specific prompting yields only six target matches across 54 counted attempts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。