错误的设计意图会干扰代码生成,比不加意图更糟。
Wrong Design Intent Is Worse Than Never Conditioning: A Derangement-Control Diagnosis of Header Conditioning in CAD Program Completion
- 用错的头信息干扰模型生成,比没加头还差。
- 3个种子下,错误头使生成正确率低于无训练基线。
- 模型依赖学到的关联,而非头信息本身。
微调代码大模型通常依赖设计意图说明进行条件生成,但错误意图(而非缺失意图)的影响尚未被检验,且评估指标常与信号定义方式一致。本文研究在Qwen2.5-Coder-1.5B上对CadQuery风格程序进行LoRA微调的CADCON任务,其头信息包含五个特征,在40%前缀长度和三个随机种子下,对400个去重测试样本(覆盖11类意图)进行评估。使用与头信息提取器无代码共享的可执行几何断言作为评价标准。结果显示:在3/3种子下,语义错误头使程序级正确率低于未训练基线;在298个独立模型输入中,3/3词元种子和2/3文本种子均出现下降。错误头的可执行性未降低。通过置换控制实验(每程序替换为另一程序的头信息),保持头分布不变但破坏其与程序的相关性,发现标准模型在正确/错误头下的表现差异为0.124/0.241/0.230,而控制组为-0.006/+0.016/-0.003,交互效应显著(p ≤ 5.9e-7)。因此模型对头信息是否正确具有学习到的敏感性,而非基线水平问题。当真实意图不存在某特征时,标准模型在错误头命名该特征时反而更常生成;控制组则无此现象。真实标签得分仅0.567,需以此为基准判断表现。错误设计意图并非中性,而是主动误导生成过程。
原文摘要 · Abstract (English)
Fine-tuned code LLMs are routinely conditioned on a design-intent specification, but the correctness axis of such a signal -- a wrong intent rather than an absent one -- has not been tested, and the benefit of conditioning is usually scored with the same detector that defines the signal. We study CADCON, a five-feature design-intent header prepended to CadQuery-style programs during LoRA fine-tuning of Qwen2.5-Coder-1.5B, scoring adherence with executable geometric assertions that share no code with the header-defining extractor. On a pre-registered sample of 400 deduplicated held-out programs stratified over eleven intent profiles, at 40% prefix and three seeds, a semantically wrong header degrades adherence below the never-header-trained baseline on 3/3 seeds under both tokenizations at the program level, and on 3/3 token and 2/3 text seeds at the 298 distinct model inputs they present. Wrong-header executability is not depressed relative to that baseline. A derangement control, retrained so every program receives another program's header -- holding the header marginal fixed while destroying its correlation with the program -- saw the same programs, indices and wrong headers. Its correct-to-wrong change is -0.006/+0.016/-0.003 against 0.124/0.241/0.230 for the standard model, and the interaction is significant on 3/3 seeds (p <= 5.9e-7), so the model's sensitivity to whether the header is right or wrong requires the learned mapping. The control sits below the baseline by the same margin under a correct as under a wrong header, so we claim that sensitivity and not the below-baseline level. On features the true intent lacks, the standard model realizes a feature far more often when the wrong header names it; the control does not. Ground truth itself scores only 0.567 here, the scale on which arm levels should be read. Wrong design intent is not inert: it actively misdirects generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。