用论坛数据构建60万条数学竞赛题库，实现模型训练与抗污染评估。
Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation
- 从AoPS论坛自动提取高质量数学习题与解答对。
- 在60万条数据上微调后，模型多基准表现提升显著。
- 构建带时间戳的动态评测集，揭示模型能力随时间退化。
大型语言模型（LLMs）在解决数学奥林匹克问题方面展现出潜力，但其训练与评估受限于高质量数据集规模小、人工标注成本高。现有评测基准还存在数据污染问题，导致结果不可靠。本文提出一种自动化流水线，利用以数学竞赛题为主的Art of Problem Solving（AoPS）论坛资源，通过开源LLM自动提取问答对，构建了包含超过60万条高质量问答对的AoPS-Instruct数据集。实验表明，在该数据集上微调的模型在多个推理基准上表现更优。此外，我们建立了自动化的动态评测流程，生成具有时间戳的LiveAoPSBench，基于最新论坛内容持续更新，提供抗污染的评测环境。值得注意的是，我们观察到模型性能随时间明显下降，暗示其在旧例题上的成功可能源于预训练暴露而非真正推理能力。本工作为高级数学推理提供了可扩展的数据构建与评估方案，揭示了当前模型在该领域的实际能力边界。相关代码与基准已公开于https://github.com/DSL-Lab/aops。
原文摘要 · Abstract (English)
Advances in Large Language Models (LLMs) have sparked interest in their ability to solve Olympiad-level math problems. However, the training and evaluation of these models are constrained by the limited size and quality of available datasets, as creating large-scale data for such advanced problems requires extensive effort from human experts. In addition, current benchmarks are prone to contamination, leading to unreliable evaluations. In this paper, we present an automated pipeline that leverages the rich resources of the Art of Problem Solving (AoPS) forum, which predominantly features Olympiad-level problems and community-driven solutions. Using open-source LLMs, we develop a method to extract question-answer pairs from the forum, resulting in AoPS-Instruct, a dataset of more than 600,000 high-quality QA pairs. Our experiments demonstrate that fine-tuning LLMs on AoPS-Instruct improves their reasoning abilities across various benchmarks. Moreover, we build an automatic pipeline that introduces LiveAoPSBench, an evolving evaluation set with timestamps, derived from the latest forum data, providing a contamination-resistant benchmark for assessing LLM performance. Notably, we observe a significant decline in LLM performance over time, suggesting their success on older examples may stem from pre-training exposure rather than true reasoning ability. Our work presents a scalable approach to creating and maintaining large-scale, high-quality datasets for advanced math reasoning, offering valuable insights into the capabilities and limitations of LLMs in this domain. Our benchmark and code is available at https://github.com/DSL-Lab/aops
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。