用可执行规范生成无限可靠测试题,打破模型记忆困局。
VeRA: Verified Reasoning Data Augmentation at Scale
- 将题目转为带占位符的可执行模板,自动生成新题
- 自动创建海量带真实答案的难题,成本接近零
- 适合研究者构建抗记忆、可扩展的评测体系
当前多数评估方案存在静态缺陷:相同问题反复使用,导致模型可记忆、格式钻空子,最终评估失效。为衡量真实人工智能进展,需从设计上保证评估鲁棒性。为此,我们提出VeRA(Verified Reasoning Data Augmentation),一种将基准问题转化为可执行规范的框架,包含:(i) 带占位符的自然语言模板,(ii) 生成有效配置的连贯生成器,(iii) 确定性验证器,用于校验参数并计算正确答案。仅需一个种子问题,VeRA即可自动生成无限数量、带可靠标签的验证变体,边际成本几乎为零,无需人工干预。该框架支持两种模式:VeRA-E(等价)重构问题但保持逻辑一致,用于检测记忆现象;VeRA-H(加固)系统提升复杂度,仍可验证,实现人类无法参与的高难度任务生成与标注。对16个前沿模型评估发现:(i) VeRA-E显著提升评估质量,揭示数据污染模式;(ii) VeRA-H实现无人工干预的难例生成与标签;(iii) VeRA确立了验证基准作为通用范式。它将基准从耗尽型静态对象转变为按需生成、持续验证的动态系统,大幅提升评估鲁棒性与成本效益。未来,任何可验证领域都可借助VeRA无限扩展评估。代码与数据集已开源,以推动后续研究。
原文摘要 · Abstract (English)
The main issue with most evaluation schemes today is their "static" nature: the same problems are reused repeatedly, allowing for memorization, format exploitation, and eventual saturation. To measure genuine AI progress, we need evaluation that is robust by construction, not by post-hoc detection. In response, we propose VeRA (Verified Reasoning Data Augmentation), a framework that converts benchmark problems into executable specifications, comprising (i) a natural language template with placeholder slots, (ii) a coherent generator that samples valid configurations, and (iii) a deterministic verifier that validates parameters and calculates the corresponding correct answers for each configuration. From a single seed problem, VeRA automatically creates unlimited verified variants with reliable labels at near-zero marginal cost without human involvement. VeRA operates in two complementary modes. VeRA-E (equivalent) rewrites problems while keeping the underlying logic intact, useful for detecting memorization versus genuine reasoning. VeRA-H (hardened) systematically increases complexity while remaining verifiable, enabling reliable creation and labelling of fresh difficult tasks at the boundary of intelligence. Evaluating 16 frontier models with VeRA, we find: (i) VeRA-E improves evaluation quality and reveals contamination patterns. (ii) VeRA-H enables human-free generation of hard tasks with reliable labels. (iii) VeRA establishes verified benchmarks as a general paradigm. VeRA reconceptualizes benchmarks from static objects used until exhausted, to executable specifications generating fresh, verified instances on demand, enhancing robustness and cost-effectiveness for evaluation. With VeRA, we envision that evaluation in any verifiable domain can scale indefinitely without sacrificing label integrity. To stimulate future research, we have open-sourced all code and datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。