用结构化上下文提升自然语言转权限代码的生成质量与可靠性。
Context-Aware Distillation and Ablation for Text2DSL
- 用语法、API和标识符词汇表构建上下文，替代传统提示生成。
- 生成10073条语义正确且99.7%可运行的权限规则对，验证率超99%。
- 词汇表是关键，能显著提升代码语义质量和结构正确性。
本文在先前工作基础上，沿两个互补方向拓展自然语言到领域特定语言（DSL）代码的自动生成。首先，采用上下文感知蒸馏替代仅依赖提示的合成生成：教师大模型（DeepSeek-V4-Flash）在明确定义的结构化上下文中运行，包含BNF语法、API规范和封闭标识符词汇表；生成语料经两级验证管道——通过esprima进行AST校验，以及通过生产环境polkitd守护进程和pkcheck客户端的运行时测试。该方法将经验证的PolkitBench数据集规模从4,204扩大至10,073对自然语言-权限规则，保持100.0%的AST有效性和99.7%的运行通过率。其次，对先前研究中提出的结构化上下文进行了组件级因子消融实验，在新数据集上评估八种条件（C0-C7），使用GigaChat-10B-A1.8B模型。主要发现包括：(i) 新更难的数据集使基线模式性能显著下降（语法有效率97.6%→58.5%，综合得分0.482→0.252），而上下文增强模式仅轻微下降（语法98.6%→97.4%，综合得分0.801→0.750），证明结构化上下文是核心支撑机制而非装饰性改进；(ii) 全上下文条件C7在所有指标上最优，最强部分条件（C5=BNF+词汇表，C6=API+词汇表）均包含词汇表；(iii) 基于Shapley分解分析，词汇表对语义质量影响最大（综合得分+0.198），而API和BNF分别对结构正确性贡献最大（分别提升24.7和22.3个百分点）。
原文摘要 · Abstract (English)
We extend our prior work on Text2DSL automatic generation of domain-specific language (DSL) code from natural language descriptions along two complementary axes. First, we replace prompt-only synthetic generation with context-aware distillation, in which a teacher large language model (DeepSeek-V4-Flash) operates under an explicitly defined structured context comprising a BNF grammar, an API specification, and a closed identifier vocabulary; the resulting corpus is verified by a two-tier pipeline combining AST validation through esprima and runtime acceptance through the production polkitd daemon and the pkcheck client. This scales the verified PolkitBench corpus from 4,204 to 10,073 natural-language-to-Polkit-rule pairs at 100.0% AST validity and 99.7% runtime pass rate. Second, we conduct the per-component factorial ablation of structured context that was identified as future work in the precursor study: eight conditions C0-C7 are evaluated on GigaChat-10B-A1.8B with the new corpus. Three findings emerge. (i) The new harder corpus collapses the baseline mode (Syntax Valid 97.6% -> 58.5%, Combined Score 0.482 -> 0.252), whereas the context-enhanced mode degrades only marginally (Syntax 98.6% -> 97.4%, Combined 0.801 -> 0.750), confirming that structured context is not a cosmetic improvement but a load-bearing mechanism. (ii) The best absolute condition is the full context C7 across all metrics, while the strongest partial conditions (C5 = BNF + Vocabulary, C6 = API + Vocabulary) both contain the vocabulary. (iii) A Shapley-style decomposition assigns the largest semantic-quality effect to the vocabulary (Combined +0.198), the largest structural-validity effects to API (+24.7 pp) and BNF (+22.3 pp).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。