让大模型在推理时动态构建新概念，提升组合推理能力
Recursive Concept Evolution for Compositional Reasoning in Large Language Models
- 通过动态生成低秩子空间扩展模型内部表征
- 在多个基准上实现12-18点性能提升
- 适合需要深度抽象推理的科研与教育场景
大语言模型在多项复杂推理任务中表现优异，但在需要组合推理的基准测试（如ARC-AGI-2、GPQA、MATH、BBH和HLE）上准确率显著下降。现有方法通过思维链提示、自一致性或强化学习扩展词元级搜索，但未改变模型的隐含表征空间。当所需抽象未编码于该空间时，性能急剧崩溃。我们提出递归概念演化（RCE）框架，使预训练语言模型在推理过程中可修改其内部表征几何结构。RCE引入动态生成的低秩概念子空间，在检测到表征不足时触发，通过最小描述长度准则选择，协同时合并，经约束优化整合以保持稳定性。该过程使模型能构建新抽象而非重组旧抽象。我们将RCE集成至Mistral-7B，并在组合推理基准上评估，结果显示在ARC-AGI-2上提升12-18点，在GPQA和BBH上提升8-14点，在MATH和HLE上一致降低深度诱导错误。
原文摘要 · Abstract (English)
Large language models achieve strong performance on many complex reasoning tasks, yet their accuracy degrades sharply on benchmarks that require compositional reasoning, including ARC-AGI-2, GPQA, MATH, BBH, and HLE. Existing methods improve reasoning by expanding token-level search through chain-of-thought prompting, self-consistency, or reinforcement learning, but they leave the model's latent representation space fixed. When the required abstraction is not already encoded in this space, performance collapses. We propose Recursive Concept Evolution (RCE), a framework that enables pretrained language models to modify their internal representation geometry during inference. RCE introduces dynamically generated low-rank concept subspaces that are spawned when representational inadequacy is detected, selected through a minimum description length criterion, merged when synergistic, and consolidated via constrained optimization to preserve stability. This process allows the model to construct new abstractions rather than recombining existing ones. We integrate RCE with Mistral-7B and evaluate it across compositional reasoning benchmarks. RCE yields 12-18 point gains on ARC-AGI-2, 8-14 point improvements on GPQA and BBH, and consistent reductions in depth-induced error on MATH and HLE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。