arXiv:2503.13447cs.CLcs.AI2025-03ACL被引 13

让大模型在推理时动态选择最优思维策略,提升复杂任务表现。

MetaScale: Test-Time Scaling with Evolving Meta-Thoughts

  • 通过元思维池与带探索的强化学习,动态生成适配任务的推理策略。
  • 在Arena-Hard上使GPT-4o胜率提升11%,优于o1-mini 0.9%。
  • 适合需要高可靠性推理的复杂任务场景,如科研、决策支持。

大型语言模型在复杂推理中常依赖训练数据中的模式匹配,而非主动选择最优认知策略。现有方法采用固定认知结构,虽在特定任务有效,但缺乏跨场景适应性。为此,我们提出MetaScale——一种基于元思维的测试时扩展框架,能为每项任务动态生成适配的思考策略。该框架初始化候选元思维池,利用带置信上界的多臂赌博机算法,结合奖励模型迭代选择并评估策略;同时引入遗传算法演化高回报元思维,持续优化策略池。实验表明,MetaScale在多项任务中显著提升准确率与泛化能力:在Arena-Hard上使GPT-4o胜率提升11%,超过o1-mini 0.9%(风格控制条件下);且随采样预算增加,性能增长更显著,生成响应更具结构化与专家级特征。

原文摘要 · Abstract (English)

One critical challenge for large language models (LLMs) for making complex reasoning is their reliance on matching reasoning patterns from training data, instead of proactively selecting the most appropriate cognitive strategy to solve a given task. Existing approaches impose fixed cognitive structures that enhance performance in specific tasks but lack adaptability across diverse scenarios. To address this limitation, we introduce METASCALE, a test-time scaling framework based on meta-thoughts -- adaptive thinking strategies tailored to each task. METASCALE initializes a pool of candidate meta-thoughts, then iteratively selects and evaluates them using a multi-armed bandit algorithm with upper confidence bound selection, guided by a reward model. To further enhance adaptability, a genetic algorithm evolves high-reward meta-thoughts, refining and extending the strategy pool over time. By dynamically proposing and optimizing meta-thoughts at inference time, METASCALE improves both accuracy and generalization across a wide range of tasks. Experimental results demonstrate that MetaScale consistently outperforms standard inference approaches, achieving an 11% performance gain in win rate on Arena-Hard for GPT-4o, surpassing o1-mini by 0.9% under style control. Notably, METASCALE scales more effectively with increasing sampling budgets and produces more structured, expert-level responses.

推理增强元思维大模型优化测试时扩展

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。