构建竞赛编程题相似性识别基准，检验模型真正理解算法本质
AlgoSimBench: Identifying Algorithmically Similar Problems for Competitive Programming
- 设计对抗性多选题，逼迫模型依赖算法逻辑而非文字表面
- 现有大模型在该任务上准确率普遍偏低，平均仅提升9%
- 适合研究算法推理、代码检索与大模型评估的学者使用
近期增强推理能力的大语言模型在解决复杂编程问题上表现良好，但其推理能力是否能泛化到识别算法相似问题（ASPs）尚不明确。本文提出 AlgoSimBench，一个包含402道多选题的基准测试集，采用对抗性设计：每道参考题配有一道算法相似题和三道语义相近但算法不同的干扰项。此设计迫使模型依赖算法层面的推理，而非表层文本线索。评估显示，当前大模型在此设置下表现持续不佳。为弥补这一差距，我们提出尝试解法匹配（ASM），利用大模型生成的解法尝试来评估相似性，使各模型平均准确率提升9%。此外，结合BM25，ASM在代码检索任务中比当前最优嵌入模型再提升11.8%。AlgoSimBench为未来大模型与检索方法的研究提供了具有挑战性的测试平台。
原文摘要 · Abstract (English)
Recent reasoning-enhanced Large Language Models (LLMs) have achieved promising results in solving complex competitive programming problems. However, it remains unclear whether these reasoning abilities generalize to relevant tasks, like identifying algorithmically similar problems (ASPs). We introduce AlgoSimBench, a benchmark of 402 multiple-choice questions curated in an adversarial setting: each given reference problem is paired with one algorithmically similar problem and three distractors that are semantically close but algorithmically dissimilar. This design forces models to rely on algorithmic reasoning rather than superficial textual cues. Our evaluation shows that LLMs consistently struggle under this setting. To address this gap, we propose Attempted Solution Matching (ASM), which leverages LLM-generated solution attempts to assess similarity, yielding an average accuracy improvement of 9% across models. Beyond LLM evaluation, AlgoSimBench also probes code retrieval methods; when combined with BM25, ASM achieves an additional 11.8% gain over state-of-the-art embedding models. AlgoSimBench offers a challenging testbed that facilitates future studies on LLMs and retrieval methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。