用自路由张量适配器实现高效多领域视觉模型适应
Self-Routed Tensor Adapters for Parameter-Efficient Universal Visual Adaptation
- 输入自动生成路由权重，动态融合共享核心块实现个性化适配
- 在5个跨域分类任务上达到媲美甚至更优的准确率，参数量减少超70%
- 适合需要轻量化多领域迁移的视觉模型部署场景
通用视觉表征需具备跨异构领域适应能力，且不将知识碎片化为特定领域模块。参数高效微调可高效适配冻结的视觉基础模型，但标准低秩适配器对所有输入使用固定子空间，当领域在风格、背景和语义上下文上差异显著时存在局限。基于MoE的适配器通过多专家路径提升专业化程度，但通常依赖外部路由网络和大型专家库，增加参数并分离路由与适配过程。本文提出自路由张量适配器（SRTA），一种紧凑的多领域视觉适配框架。SRTA将每个输入投影至低秩空间，利用可学习的领域矩阵从该表示中计算路由权重，并用这些权重混合共享的Tucker核心切片，生成样本相关的适配矩阵，无需外部门控网络，使共享视觉因子得以复用的同时支持领域感知专业化。为强化路径学习，引入逐层深度加权路由目标，监督各适配层的路由决策。在五个异构多领域视觉分类基准上，SRTA在平均准确率上达到竞争性或略优表现，同时显著减少可训练参数。在秩为64时，四领域设置下仅使用277万参数（对比MoLoRA的952万），六领域设置下为300万参数（对比1431万）。总体而言，SRTA为视觉基础模型向通用多领域表征的适应提供了高效的精度-参数权衡。
原文摘要 · Abstract (English)
Universal visual representations require adaptation mechanisms that adapt across heterogeneous domains without fragmenting knowledge into domain-specific modules. Parameter-efficient fine-tuning adapts frozen visual foundation models efficiently, but standard low-rank adapters use a fixed subspace for all inputs, which can be restrictive when domains differ in style, background, and semantic context. MoE-based adapters improve specialization through multiple expert pathways, but often rely on external routers and large expert banks, adding parameters and separating routing from adaptation. We propose \textbf{Self-Routed Tensor Adapters}, a compact framework for multi-domain visual adaptation. SRTA projects each input into a low-rank space, computes routing weights from this representation using a learnable domain matrix, and uses these weights to blend slices of a shared Tucker core. This produces a sample-specific adaptation matrix without an external gating network, allowing shared visual factors to be reused while supporting domain-aware specialization. To strengthen pathway learning, we introduce a progressive depth-weighted routing objective that supervises routing decisions across adapter layers. Across five heterogeneous multi-domain visual classification benchmarks, SRTA achieves competitive or slightly stronger average accuracy than MoE-style PEFT baselines while using substantially fewer trainable parameters. At rank 64, SRTA uses 2.77M parameters in the 4-domain setting compared with 9.52M for MoLoRA, and 3.00M in the 6-domain setting compared with 14.31M. Overall, SRTA offers an effective accuracy-parameter trade-off for adapting visual foundation models toward universal multi-domain representations. \href{https://github.com/surajyadav-research/SRTA}{GitHub}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。