评测大模型在基于属性测试中的推理能力，要求从文档推断规则并设计精准测试输入。
PBT-Bench: Benchmarking AI Agents on Property-Based Testing
- 构建100个真实Python库的属性测试题，需从文档推断语义不变式并设计精准输入策略。
- 模型在引导式提示下最高召回率83.4%，普通提示下最高76.7%，结构化提示提升中等模型表现。
- 最难问题呈现模型特异性，不同架构在不同问题上失败，无模型能全面覆盖所有缺陷。
现有代码评测仅关注能否生成复现已知漏洞的测试或修复方案，无法衡量属性测试的核心能力：从文档中推导语义不变式，并设计足够精确的输入生成策略以触发潜在缺陷。本文提出PBT-Bench，包含40个真实Python库的100个精心设计的属性测试问题，共注入365个语义漏洞（平均每题3.65个），这些漏洞在默认随机输入下几乎无法触发。模型需阅读文档、识别关键不变式，并编写Hypothesis @given策略，集中探测触发区域。漏洞按难度分为三级（L1-L3），涵盖单约束边界错误到跨函数状态协议违规。评估八种主流LLM在两种提示方式（开放基线与显式Hypothesis支架）下的表现，每配置独立运行三次。在PBT引导提示下，模型漏洞召回率介于42.1%至83.4%之间；开放基线为31.4%至76.7%。结构化提示使中等能力模型提升超20个百分点，但对最强模型增益有限，且两个模型出现性能下降，表明提示可能干扰某些模型行为而非增强其能力。最困难的问题呈现显著模型特异性：不同架构在不同问题上失败，留下持续存在的空白，单一模型无法填补。
原文摘要 · Abstract (English)
Existing code benchmarks measure whether an agent can produce any test that reproduces a known bug, or whether it can produce a patch that fixes a described issue. Neither isolates the distinct skill of property-based testing: deriving a semantic invariant from documentation, and then constructing an input-generation strategy precise enough to make a random search reveal the violation. We introduce PBT-Bench, a benchmark of 100 curated property-based testing problems across 40 real Python libraries. Each problem injects one or more semantic bugs (365 in total, mean 3.65 per problem) designed so that default-strategy random inputs almost never trigger them; the agent must read the library's documentation, identify the relevant invariant, and specify a Hypothesis @given strategy that concentrates mass in the trigger region. Bugs are stratified across three difficulty levels (L1-L3) spanning single-constraint boundary bugs to stateful, cross-function protocol violations. We evaluate eight contemporary LLMs under two prompting regimes (open-ended baseline vs. explicit Hypothesis scaffolding) for three independent runs per configuration. Bug recall under the PBT-guided prompt ranges from 42.1% to 83.4% across models; under the open-ended baseline, from 31.4% to 76.7%. Hypothesis scaffolding lifts mid-capability models by over 20 percentage points, but yields smaller gains for the strongest models, with two exceptions showing degradation, suggesting the structured prompt can interfere with certain model behaviours rather than complementing them. The hardest bugs prove model-specific: different architectures fail on different problems, leaving persistent gaps that no single model closes. We release the benchmark, harness, and full evaluation corpus to support downstream work on documentation-grounded semantic reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。