用LLM生成并筛选多样化解法，显著提升抽象推理能力
Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspective
- 通过数据增强与深度优先搜索生成多样高概率解法
- 在公开ARC-AGI测试集上达到71.6%准确率（286.5/400）
- 方法透明可复现，单任务推理成本仅约2美分
抽象与推理语料库（ARC-AGI）对大语言模型（LLMs）构成重大挑战，暴露其在抽象推理方面的局限性。本文在训练、生成和评分阶段均引入任务特定的数据增强，并采用深度优先搜索算法生成多样且高概率的候选解。此外，我们不仅将LLM用于生成，还将其作为评分器，利用输出概率筛选最具前景的解。该方法在公开的ARC-AGI评估集上取得71.6%的得分（286.5/400），为当前公开方法中的最佳表现。尽管同期封闭源代码工作报告了更高分数，但本方法在透明性、可复现性和极低推理成本方面具有优势，单任务平均成本仅为2美分左右（基于36美分/小时的Nvidia 4090 GPU价格估算）。
原文摘要 · Abstract (English)
The Abstraction and Reasoning Corpus (ARC-AGI) poses a significant challenge for large language models (LLMs), exposing limitations in their abstract reasoning abilities. In this work, we leverage task-specific data augmentations throughout the training, generation, and scoring phases, and employ a depth-first search algorithm to generate diverse, high-probability candidate solutions. Furthermore, we utilize the LLM not only as a generator but also as a scorer, using its output probabilities to select the most promising solutions. Our method achieves a score of 71.6% (286.5/400 solved tasks) on the public ARC-AGI evaluation set, demonstrating state-of-the-art performance among publicly available approaches. While concurrent closed-source work has reported higher scores, our method distinguishes itself through its transparency, reproducibility, and remarkably low inference cost, averaging only around 2ct per task on readily available hardware (we assume a price of 36ct/hour for a Nvidia 4090 GPU).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。