用验证器筛选高质量答案，实现大模型持续自我优化。
Continuous Self-Improvement of Large Language Models by Test-time Training with Verifier-Driven Sample Selection
- 通过验证器评分筛选高置信度生成结果，用于测试时训练。
- 在三个基准上相对基线提升最高达32.29%，优于无测试时训练的验证方法。
- 仅微调低秩LoRA参数，适合实时部署与资源受限场景。
将预训练语言模型适应于未标注、分布外的数据是关键挑战，因模型在结构新颖的推理任务上常表现不佳。本文提出VDS-TTT框架——基于验证器的测试时训练样本选择，以高效应对该问题。对于每个输入查询，大模型生成N个候选答案；验证器为每个答案分配可靠性分数，仅选择置信度最高且超过阈值的响应与其查询配对，用于测试时微调。仅更新低秩LoRA适配器参数，确保高效性与快速收敛。所提自监督框架首次实现由验证器驱动的测试时训练数据合成，支持模型持续自我改进。在三个多样化基准和三种先进大模型上的实验表明，相比基线模型最高提升32.29%，比无测试时训练的验证方法提升6.66%，验证了其有效性与效率。
原文摘要 · Abstract (English)
Learning to adapt pretrained language models to unlabeled, out-of-distribution data is a critical challenge, as models often falter on structurally novel reasoning tasks even while excelling within their training distribution. We introduce a new framework called VDS-TTT - Verifier-Driven Sample Selection for Test-Time Training to efficiently address this. We use a learned verifier to score a pool of generated responses and select only from high ranking pseudo-labeled examples for fine-tuned adaptation. Specifically, for each input query our LLM generates N candidate answers; the verifier assigns a reliability score to each, and the response with the highest confidence and above a fixed threshold is paired with its query for test-time training. We fine-tune only low-rank LoRA adapter parameters, ensuring adaptation efficiency and fast convergence. Our proposed self-supervised framework is the first to synthesize verifier driven test-time training data for continuous self-improvement of the model. Experiments across three diverse benchmarks and three state-of-the-art LLMs demonstrate that VDS-TTT yields up to a 32.29% relative improvement over the base model and a 6.66% gain compared to verifier-based methods without test-time training, highlighting its effectiveness and efficiency for on-the-fly large language model adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。