小模型生成代码出错,后处理修复无效,唯有特定提取方式能提升准确率。
Selection Without Signal, Recovery Through Expression: A Measurement Study of Post-Hoc Falsification Operators for Frozen Small Code Models
- 测试26种后处理方法,发现均不如基础采样法(BoN)
- 新提取方法M1使小模型在HumanEval+上多解12题,显著提效
- 适合关注隐私与本地部署的开发者,尤其需优化生成质量者
冻结的小型代码模型(≤1.5B参数,无需微调即可本地运行)适用于离线和隐私受限场景,但常生成看似合理实则错误的程序。一种自然解决方案是后处理操作:在不重新训练的前提下,对模型输出进行选择、验证、修复或重处理。从波普尔主义出发,即用严苛测试筛选候选,保留通过者。本研究在无泄漏、匹配计算的确定性执行环境下,评估26种语义级后处理操作(包括选择、验证、修复、剔除、组合、可靠否决、生成条件化),结果表明,在测试的多个数据集与基准上,所有方法均未超越Best-of-N(BoN)的保留准确率。负面结果源于三重机制障碍:覆盖墙(深度困难任务无法通过增加采样缓解)、能力剪刀(优秀生成器产生的可识别错误极少)、近空共识陷阱(可见通过但隐藏错误的多数情况与正确替代方案几乎不共现)。分布无关的无害边界显示,若零观测伤害,则需n≥45才能保证危害率≤α。唯有两个非语义空间操作有效:表达层恢复(M1)首次实现准确率提升,能回收标准提取器丢弃的正确程序(鲁棒提取与公开测试签名对齐),且无损害(b10=0),在DeepSeek-Coder-1.3B上使HumanEval+多解12题(p=2.4e-4);自适应共识早停(ACE)实现约19%算力节省且零损害。M1与负结果在HumanEval+与MBPP+跨三个模型组复现。结论:应先优化评测框架与覆盖率,再责难语义后处理推理。
原文摘要 · Abstract (English)
Frozen small code models (<=1.5B parameters, run locally without fine-tuning) suit offline and privacy-constrained use, but often emit plausible-but-wrong programs. A natural remedy is a post-hoc operator that selects, verifies, repairs, or re-processes the model's samples without retraining; in principled form it is Popperian: attack each candidate with a severe test, keep what survives. We measure whether such operators help. Under one deterministic execution oracle and a leakage-free, matched-compute protocol, 26 semantic post-hoc operators (selection, verification, repair, elimination, portfolios, sound vetoes, generation conditioning) are evaluated against Best-of-N (BoN); on the cells and benchmarks tested, none improves held-out accuracy over BoN. The negative is mechanistic: a coverage wall (systematic hard-task failures deeper sampling does not rescue), a capability scissors (a competent generator leaves almost no discriminable error among visible-test passers), and a near-empty consensus trap (the visible-pass-but-hidden-wrong majority a leakage-free selector needs rarely co-occurs with a correct alternative). A distribution-free do-no-harm bound cannot certify a harm rate <=alpha at zero observed harm unless n>=45. Two operators help on a different axis, outside the semantic output space. An expression-layer recovery (M1), the only accuracy gain here, recovers correct programs the standard extractor discards (robust extraction and public-test signature alignment); it does no harm (b10=0), is leakage-free, and lifts DeepSeek-Coder-1.3B by +12 tasks on HumanEval+ (p=2.4e-4). An adaptive consensus early-stop (ACE) is a calibrated compute-saving control (~19% saving, zero harm). M1 and the selection negative replicate on HumanEval+ and MBPP+ across three model cells. The lesson: fix the harness and measure coverage before blaming semantic post-hoc reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。