用对抗强化学习训练大模型提炼编程问题本质模式。
AR$^2$: Adversarial Reinforcement Learning for Abstract Reasoning in Large Language Models
- 教师模型将核心问题转化为复杂叙事，学生模型从中提取计算本质。
- 在未见过的难题上准确率显著提升，验证抽象能力重要性。
- 适合想提升大模型泛化能力的研究者与开发者参考。
抽象——从复杂问题描述中识别并提炼出核心计算模式的能力——是计算机科学的基础技能，对人类解题者和面向代码的大语言模型（LLMs）都至关重要。尽管近期基于强化学习（RL）训练的LLMs在代码生成方面取得进展，但多数方法仍聚焦于表面模式识别，缺乏对抽象能力的显式训练。本文提出AR²（Adversarial Reinforcement Learning for Abstract Reasoning），一种专为增强LLM抽象能力设计的新框架。该框架通过教师模型将核心问题转化为富含叙事、具有挑战性的描述，同时训练学生编码模型在不改变底层逻辑的前提下，从这些复杂叙述中提取出其本质计算内核。实验结果表明，使用AR²训练的学生模型在未见过的高难度编程任务上准确率显著提升，凸显了抽象能力对提升大模型泛化性能的关键作用。
原文摘要 · Abstract (English)
Abstraction--the ability to recognize and distill essential computational patterns from complex problem statements--is a foundational skill in computer science, critical both for human problem-solvers and coding-oriented large language models (LLMs). Despite recent advances in training LLMs for code generation using reinforcement learning (RL), most existing approaches focus primarily on superficial pattern recognition, overlooking explicit training for abstraction. In this study, we propose AR$^2$ (Adversarial Reinforcement Learning for Abstract Reasoning), a novel framework explicitly designed to enhance the abstraction abilities of LLMs. AR$^2$ employs a teacher model to transform kernel problems into narrative-rich, challenging descriptions without changing their fundamental logic. Simultaneously, a student coding model is trained to solve these complex narrative problems by extracting their underlying computational kernels. Experimental results demonstrate that AR$^2$ substantially improves the student model's accuracy on previously unseen, challenging programming tasks, underscoring abstraction as a key skill for enhancing LLM generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。