提出无需答案的模型准入机制,提升大模型优化建模的可靠性。
Admission Without Answers: Label-Free Certification and Experience Learning for LLM-Based Optimization Modeling

- 基于行为证据交叉验证,用校准阈值判断模型是否可接纳。
- 在300个无标签问题流上,准入准确率达0.927,误入率降低8倍。
- 适合需要高可信度建模的工业场景,尤其适用于无标注数据环境。
基于经验学习的优化建模代理通过存储已验证技能来提升性能,但现有方法依赖已知答案进行知识确认,而真实任务流中并无此类反馈。自然的无标签替代方案不可靠:在300个无标签问题流上,接受所有可执行模型会导致约四分之一被污染;单一实例一致仅匹配一个值,却忽略其他差异。本文提出AdmitOR,一种基于校准外部行为证据的准入门控机制。从三个模型族、提示策略与求解器堆栈中选取候选模型,在参数域重采样实例上运行;通过跨家族值函数轨迹的一致性构建共现团,再以校准阈值输出接受、放弃或升级。预注册的错误发现率在校准数据上成立,但在真实流中失败。我们完整报告此负面结果,并发现多数失败源于基准文本未能忠实编码其标注实例。在最先进的技能学习器内对比四个准入判别器,AdmitOR将准入精确率提升至0.927,远超多数投票(0.871)和执行成功(0.726),误入次数分别减少3.1倍和8.0倍。其知识库最小,且在五个公开基准上取得最高宏观准确率58.4,优于多数投票(54.8)与真值标注库(53.9)。相对于多数投票的3.5点提升经配对自助法验证,且在主机侧异常校正后仍成立。据我们所知,AdmitOR是首个围绕显式校准错误发现目标设计的无标签准入机制。转移失败揭示了将其扩展至真实流的必要条件。
原文摘要 · Abstract (English)
Experience-learning agents for optimization modeling improve by storing verified skills, but existing learners admit knowledge by checking against known answers, which real ticket streams do not provide. The natural label-free alternatives are unreliable: on a 300-problem label-blind stream, admitting every executable model poisons roughly one admission in four, while single-instance agreement accepts models that match at one value but differ elsewhere. We propose AdmitOR, an admission gate built on calibrated external behavioral evidence. Candidates from three model families, prompting strategies, and solver stacks are run on instances resampled from an extracted parameter domain; agreement across the resulting value-function traces is summarized by a cross-family clique, and a calibrated threshold returns accept, abstain, or escalate. The preregistered false-discovery criterion holds on calibration data but not on the wild stream. We report this negative result in full and trace most failures to benchmark texts that do not faithfully encode their labeled instances. Comparing four admission judges on one collection of logs inside a state-of-the-art skill learner, AdmitOR raises admission precision to 0.927, against 0.871 for majority vote and 0.726 for execution success, yielding 3.1x and 8.0x fewer poisoned admissions. Its library is the smallest and attains the highest macro accuracy across five public benchmarks, 58.4 against 54.8 for majority vote and 53.9 for the ground-truth-labeled library. The 3.5-point gain over majority vote is supported by a paired bootstrap and survives correction for a host-side anomaly. To our knowledge, AdmitOR is the first label-free admission mechanism designed around an explicitly calibrated false-discovery target. The transfer failure identifies a necessary condition for extending it to wild streams.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。