根据奖励模型动态选择最优推理计算策略,提升大模型性能。
RTTC: Reward-Guided Collaborative Test-Time Compute
- 用预训练奖励模型决定每条查询该用哪种推理增强方式。
- 在多个基准上比传统RAG或TTT准确率更高。
- 支持缓存历史查询状态,减少重复计算,适合实际部署。
测试时计算(TTC)已成为提升大语言模型推理性能的有效范式,包括测试时训练(TTT)和检索增强生成(RAG)。然而,不同查询的最佳适应策略各异,盲目应用TTC会带来显著计算开销。本文提出奖励引导的测试时计算(RTTC),通过预训练奖励模型为每条查询自适应选择最有效的TTC策略,从而在多样任务与领域中最大化下游准确率。RTTC采用分布式服务器-客户端架构,仅在必要时从远程知识库检索并本地执行RAG或轻量微调。为进一步降低冗余计算,提出查询状态缓存机制,实现检索与适配层面的历史状态高效复用。在多个LLM与基准上的大量实验表明,RTTC始终优于原始RAG或TTT,验证了自适应、奖励引导的TTC选择的必要性及可扩展高精度模型适配的潜力。
原文摘要 · Abstract (English)
Test-Time Compute (TTC) has emerged as a powerful paradigm for enhancing the performance of Large Language Models (LLMs) at inference, leveraging strategies such as Test-Time Training (TTT) and Retrieval-Augmented Generation (RAG). However, the optimal adaptation strategy varies across queries, and indiscriminate application of TTC strategy incurs substantial computational overhead. In this work, we introduce Reward-Guided Test-Time Compute (RTTC), a novel framework that adaptively selects the most effective TTC strategy for each query via a pretrained reward model, maximizing downstream accuracy across diverse domains and tasks. RTTC operates in a distributed server-client architecture, retrieving relevant samples from a remote knowledge base and applying RAG or lightweight fine-tuning on client devices only when necessary. To further mitigate redundant computation, we propose Query-State Caching, which enables the efficient reuse of historical query states at both retrieval and adaptation levels. Extensive experiments across multiple LLMs and benchmarks demonstrate that RTTC consistently achieves superior accuracy compared to vanilla RAG or TTT, validating the necessity of adaptive, reward-guided TTC selection and the potential of RTTC for scalable, high-performance language model adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。