让LoRA适配器在测试时动态选择最优组合,提升模型泛化能力。
LiST: Local-Simplex Test-Time LoRA Fusion

- 基于提示的局部单纯形搜索,动态生成样本专属融合权重。
- 在多模态与语言任务上超越静态合并与传统测试时适应方法。
- 无需更新主干或适配器,适合部署在资源受限场景。
特定任务的LoRA适配器为大语言和视觉语言模型提供了模块化定制方式。然而,现有适配器组合方法多为静态,无法根据具体测试输入自适应调整。为此,我们提出LiST——一种无标签的测试时LoRA融合框架,将现有LoRA库转化为目标条件下的局部单纯形,并在推理时搜索样本特定的融合权重。LiST通过LoRA参数锚点与提示级行为向量构建联合任务表示,检索邻近适配器作为局部搜索空间,并执行不更新主干或适配器的分支保持融合。候选权重由带有先验、几何与随机一致性约束的提示级能量函数选择,仅在通过安全接受规则后才部署;否则回退至目标条件先验。在多模态与语言基准上的实验表明,LiST优于静态LoRA合并及常规测试时适应基线,同时保持任务特异性适配器效用,并提升对未见任务的鲁棒性。
原文摘要 · Abstract (English)
Task-specific LoRA adapters offer a modular way to specialize large language and vision-language models. However, existing adapter composition methods are mostly static and cannot adapt to individual test inputs. To address these issues, we propose \textbf{LiST}, a label-free test-time LoRA fusion framework that converts an existing LoRA bank into a target-conditioned local simplex and searches sample-specific fusion weights at inference time. LiST builds joint task representations from LoRA parameter anchors and prompt-level behavior vectors, retrieves neighboring adapters as a local search space, and performs branch-preserving fusion without updating the backbone or adapters. Candidate weights are selected by a prompt-level energy with prior, geometric, and stochastic-consistency constraints, and are deployed only when they pass a safe acceptance rule. Otherwise, LiST falls back to a target-conditioned prior. Experiments on multimodal and language benchmarks show that LiST outperforms static LoRA merging and conventional test-time adaptation baselines, while preserving task-specific adapter utility and improving robustness on unseen tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。