构建首个编程竞赛检索基准，解决题目重复问题
CPRet: A Dataset, Benchmark, and Model for Retrieval in Competitive Programming
- 设计四类检索任务，结合自动爬取与人工标注数据
- 提出两个专用检索模型，准确率超基线15%以上
- 适合评估大模型编程能力或开发相似题检测工具
编程竞赛基准广泛用于编程比赛和大语言模型评估。然而，题目重复或高度相似现象日益严重，不仅影响比赛公平性，也削弱了其作为模型评估基准的有效性。本文提出相似题目检索新任务以应对该问题。由于缺乏数据和模型，该任务极具挑战。为此，我们推出CPRet——面向检索的编程竞赛基准套件，包含四类检索任务：两类代码中心任务（文本到代码、代码到代码）和两类新提出的题目中心任务（题目到重复题、简化题到完整题），数据源自自动爬取的问题-解法对与人工标注。贡献包括高质量训练数据及时间分离的测试集，确保可靠评估。此外，我们开发了两个任务专用检索器：基于新型分组InfoNCE损失的CPRetriever-Code，用于题目-代码对齐；以及针对题目级相似性识别微调的CPRetriever-Prob。两者均表现优异，已开源供本地使用。最后，我们分析LiveCodeBench发现，高相似题目会显著提高模型通过率并降低区分度，凸显未来基准需引入相似性感知评估。项目主页：https://github.com/coldchair/CPRet；在线演示：https://www.cpret.online/
原文摘要 · Abstract (English)
Competitive programming benchmarks are widely used in scenarios such as programming contests and large language model assessments. However, the growing presence of duplicate or highly similar problems raises concerns not only about competition fairness, but also about the validity of competitive programming as a benchmark for model evaluation. In this paper, we propose a new problem, similar question retrieval, to tackle this issue. Due to the lack of both data and models, solving this problem is challenging. To this end, we introduce CPRet, a retrieval-oriented benchmark suite for competitive programming, covering four retrieval tasks: two code-centric (i.e., Text-to-Code, Code-to-Code) and two newly proposed problem-centric tasks (i.e., Problem-to-Duplicate, Simplified-to-Full) built from a combination of automatically crawled problem-solution data and manually curated annotations. Our contribution includes both high-quality training data and temporally separated test sets for reliable evaluation. Besides, we further develop two task-specialized retrievers based on this dataset: CPRetriever-Code, trained with a novel Group-InfoNCE loss for problem-code alignment, and CPRetriever-Prob, fine-tuned for identifying problem-level similarity. Both models achieve strong results and are open-sourced for local use. Finally, we analyze LiveCodeBench and find that high-similarity problems inflate model pass rates and reduce differentiation, underscoring the need for similarity-aware evaluation in future benchmarks. Github: https://github.com/coldchair/CPRet Online Demo: https://www.cpret.online/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。