提出规范形式化模型与执行评判基准,量化规范确定性。
Measuring What a Specification Determines: A Formal Semantic-Block Model and an Execution-Judged Benchmark
- 用语义块结构建模规范,定义四类可机器验证的良构条件。
- 实测显示五层分解使上下文减少71%,覆盖85.5%的数据库构造分类。
- 通过独立实现者收敛度评估规范质量,适合关注规范严谨性的研究者。
本文提出一种规范的形式化语义块模型和一个基于执行判断的评测基准,用于在不依赖模型能力的前提下评估规范质量。规范被表示为包含语义块、依赖关系、块所有规则、决策点及显式开放问题的结构,并满足四项可机器检验的良构条件:无环性、单一所有权、约束支配性与完备性或歧义终止。确定性从模型论角度定义为所有符合实现的一致性,并通过独立实现者间的收敛性进行经验估计。该模型应用于包含18个块和19条依赖边的Oracle到PostgreSQL迁移规范。计算验证表明,五层分解通过依赖闭包将平均任务上下文减少约71%,覆盖研究定义的Oracle构造分类的85.5%,所有识别出的缺口均已归类处理,且未被测试的替代划分方案所优于;该结构可在引用导出的未用于原结构构建的边中以99.9百分位恢复。基准保持实现者小组固定,包含强制无规范对照组,并使用PostgreSQL 16和实时Oracle实例作为确定性执行裁判。六项设计研究(包括三个预注册操控和三个诊断分析)进一步检验规范影响。对25单元子样本的重复运行揭示经验变异性下限,中位臂间差异为14.4个百分点。结果支持确定性作为正式概念,但不支持其作为当代大模型实现者评价的独立经验质量指标。
原文摘要 · Abstract (English)
This work introduces a formal semantic-block model for specifications and an execution-judged benchmark for evaluating specification quality independently of model capability. A specification is represented as a structure comprising semantic blocks, dependency relations, block-owned rules, decision points, and explicitly open questions, subject to four machine-checkable well-formedness conditions: acyclicity, single ownership, constraint domination, and totality or ambiguity-stop. Determinacy is defined model-theoretically as agreement among all conforming implementations and is estimated empirically through convergence across independent implementers. The model is instantiated on an Oracle-to-PostgreSQL migration specification containing 18 blocks and 19 dependency edges. Computational validation shows that the five-layer decomposition reduces mean per-task context by approximately 71% through dependency closures, covers 85.5% of the study-defined Oracle construct taxonomy with all identified gaps triaged, is not Pareto-dominated by the tested alternative partitions, and is recovered at the 99.9th percentile from citation-derived edges not used to define the original structure. The benchmark keeps the implementer panel fixed, includes a mandatory no-specification control arm, and uses PostgreSQL 16 and a live Oracle instance as deterministic execution judges. Six designed studies, including three pre-registered manipulations and three diagnostic analyses, further examine specification effects. Repeated runs on a 25-unit subsample reveal an empirical variability floor with a median arm-delta spread of 14.4 percentage points. The results support determinacy as a formal concept but not as a standalone empirical quality metric for the evaluated contemporary LLM implementers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。