强制代码引用可提升检测幻觉能力，但会降低生成一致性。
Citation Discipline in Spec-Driven Development: A Cross-Model Empirical Study of Output Determinism and Automated Hallucination Detection in LLM-Generated Code
- 通过强制逐行引用需求编号，提升代码可追溯性。
- 引用条件使幻觉检测率高达86%-88%，无引用则为0%。
- 适合关注代码可信度与验证的开发团队使用。
Spec-Driven Development (SDD) 框架通过正式规范指导大语言模型生成代码，但不同框架在需求与代码间追踪机制上存在根本差异。本文通过两组受控实验，对比三种 SDD 框架：traceSDD（强制使用层级 REQ-XXX.Y.Z 标识符逐行引用）、Spec Kit（基于用户故事和验收标准的构件级追踪）以及 OpenSpec（依赖事后外部追踪图）。研究在 Claude Sonnet 4.6（N=20，4种条件，240个实现）和 GLM-5-turbo（N=50，4种条件，600个实现）上测量输出确定性（独立会话间的词法相似度）和自动化幻觉检测率（TDR）。预注册分析显示一致且跨模型重复的权衡关系：未引用条件下输出确定性显著高于引用条件（Claude: d=-0.76, p=0.003；GLM: d=-0.72, p<0.001），而仅引用条件能实现幻觉检测（TDR：Claude 86.4%，GLM 88.0%，其余为0%；所有研究中假阳性率均为0%）。traceSDD（引用）在确定性上显著优于 Spec Kit（Claude: d=0.47, p=0.049；GLM: d=0.42, p=0.003），但与 OpenSpec 无显著差异（Claude: d=0.18, p=0.44；GLM: d=0.14, p=0.32）。结果表明，引用注释以牺牲确定性换取可验证性，且该权衡在不同模型架构间具有普适性。
原文摘要 · Abstract (English)
Spec-Driven Development (SDD) frameworks guide Large Language Model (LLM)-powered code generation through formal specifications, yet they differ fundamentally in how they enforce traceability between requirements and generated code. This paper presents two controlled empirical studies comparing three SDD frameworks: $traceSDD$, which enforces mandatory per-line requirement citations using hierarchical REQ-XXX.Y.Z identifiers; $Spec Kit$, which uses artifact-level traceability through user stories and acceptance criteria; and $OpenSpec$, which relies on post-hoc external trace maps. We measure two primary outcomes across two frontier LLMs -- Claude Sonnet 4.6 (N=20, 4 conditions, 240 implementations) and GLM-5-turbo (N=50, 4 conditions, 600 implementations): $output$ $determinism$ (lexical similarity across independent LLM sessions) and $automated$ $hallucination$ $detection$ $rate$ (TDR). Our pre-registered analysis reveals a consistent, cross-model replicated trade-off: the uncited condition produces significantly higher determinism than the cited condition (Claude: $d=-0.76$, $p=0.003$; GLM: $d=-0.72$, $p<0.001$), while only the cited condition enables automated hallucination detection (TDR: Claude 86.4%, GLM 88.0%, vs 0% for all alternatives, FPR=0% across both studies). traceSDD (cited) significantly outperforms $Spec Kit$ on determinism (Claude: $d=0.47$, $p=0.049$; GLM: $d=0.42$, $p=0.003$) but not OpenSpec (Claude: $d=0.18$, $p=0.44$; GLM: $d=0.14$, $p=0.32$). These findings establish that citation annotations trade determinism for verifiability, and that this trade-off generalizes across model architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。