比较四种模型专业化方法,发现生成专用权重最省成本且可复用。
Model of Models: When Does Emitting a Specialist Beat Attending, Adapting, or Tuning?

- 通过超网络生成专用权重,避免重复推理
- 在临床分类中达到顶尖水平,且每例仅需132浮点数
- 适合需要快速响应和低成本部署的场景
面对少量示例定义的任务,如何高效定制模型?本文对比了零样本、上下文注意力、测试时梯度调整和超网络生成专用权重四种方式。在涵盖回归、生成、语言建模、强化学习及临床与基因组分类的六项任务上,保持专家模型、上下文和训练预算一致。结果显示,生成专用权重在匹配质量下成本最低:在临床少样本分类中媲美最先进的表格式模型(TabPFN),且无需每查询重读支持集;在正弦波少样本回归中,即使无测试梯度更新,也比MAML低2-3个数量级,等量训练后仍保持约30倍优势。但在高维序列建模中,其性能无法超越上下文注意力——在相同预训练预算下,单次适配器仅恢复14.0±0.9%(500万参数)或11.2±0.5%(1500万参数)的上下文增益,且LoRA秩提升仅使捕捉率从5%升至21%即趋于饱和。机制消融验证了生成的专用权重是真实任务条件化的,非记忆先验;更进一步推测其权重空间具有组合性——两权重重叠可追踪对应功能混合。最后提出可验证假说,通过任务分辨率指标界定各机制适用边界。
原文摘要 · Abstract (English)
Given a task described by a few examples, how should a model be specialized to it? Four mechanisms are available -- zero-shot, in-context attention, test-time gradient adaptation, and emitting specialist weights from a hypernetwork -- yet the operating regime of the last is rarely mapped. We run the identical four-way comparison across six tasks spanning regression, generation, language modeling, reinforcement learning, and clinical and genomic classification, holding the specialist, the context, and (where we can) the training budget fixed. The clearest wins for emission are about cost at matched quality: it ties the state-of-the-art amortized tabular model (TabPFN) on clinical few-shot classification while emitting a reusable specialist instead of re-attending the support set per query, and reaches noise-floor shape generation with a $132$-float per-instance program. On few-shot sinusoid regression it is $2$--$3$ orders of magnitude below MAML at zero test-time gradient steps -- a margin that narrows to $\sim$$30\times$ but persists once training budgets are equalized. Emission cannot match in-context attention on high-dimensional sequence modeling: under matched-budget pre-training a one-pass adapter recovers only a minority of the in-context gain ($14.0\pm0.9\%$ at $5$M, $11.2\pm0.5\%$ at $15$M), and a LoRA-rank sweep shows this shortfall is a partial capacity limit -- capture climbs from $5\%$ to $21\%$ as rank grows but plateaus far below full recovery. Mechanism ablations confirm the emitted specialist is genuinely task-conditioned, not a memorized prior; and, more speculatively, emitted specialists compose in weight space -- interpolating two of them tracks the corresponding blend of their functions. We close with a falsifiable thesis, operationalized through a per-task resolution measure, bounding when each conditioning mechanism should be preferred.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。