用冻结模型构建可训练图结构,实现高效多模型协作推理。
Dead Weights, Live Signals: Feedforward Graphs of Frozen Language Models
- 将多个冻结大模型作为节点,通过可学习投影连接在共享潜空间中。
- 仅1760万可训练参数即达87.3%的ARC-Challenge得分,超越单模型11.4个百分点。
- 无需显式监督即可实现跨层选择性路由,适合轻量级多模型集成场景。
我们提出一种前馈图架构,其中异构的冻结大型语言模型作为计算节点,通过共享连续潜空间并借助学习的线性投影进行通信。基于近期研究证明独立训练的LLM潜空间存在几何兼容性,我们将该发现从静态双模型调参扩展至端到端可训练的多节点图结构,其中投影矩阵通过残差流注入钩子进行反向传播联合优化。三个小型冻结模型(Llama-3.2-1B、Qwen2.5-1.5B、Gemma-2-2B)将输入编码至共享潜空间,其聚合信号注入两个较大冻结模型(Phi-3-mini、Mistral-7B),其表示再输入一个轻量级交叉注意力输出节点。仅1760万可训练参数(相比约120亿冻结参数),该架构在ARC-Challenge上取得87.3%、OpenBookQA上82.8%、MMLU上67.2%的准确率,分别优于最佳单个构成模型11.4、6.2、1.2个百分点,且优于参数匹配的冻结单模型学习分类器9.1、5.2、6.7个百分点。实验证明穿过多个冻结模型边界的梯度流动是可行的,输出节点在无显式监督下发展出对层2节点的选择性路由行为。
原文摘要 · Abstract (English)
We present a feedforward graph architecture in which heterogeneous frozen large language models serve as computational nodes, communicating through a shared continuous latent space via learned linear projections. Building on recent work demonstrating geometric compatibility between independently trained LLM latent spaces~\cite{armstrong2026thinking}, we extend this finding from static two-model steering to end-to-end trainable multi-node graphs, where projection matrices are optimized jointly via backpropagation through residual stream injection hooks. Three small frozen models (Llama-3.2-1B, Qwen2.5-1.5B, Gemma-2-2B) encode the input into a shared latent space whose aggregate signal is injected into two larger frozen models (Phi-3-mini, Mistral-7B), whose representations feed a lightweight cross-attention output node. With only 17.6M trainable parameters against approximately 12B frozen, the architecture achieves 87.3\% on ARC-Challenge, 82.8\% on OpenBookQA, and 67.2\% on MMLU, outperforming the best single constituent model by 11.4, 6.2, and 1.2 percentage points respectively, and outperforming parameter-matched learned classifiers on frozen single models by 9.1, 5.2, and 6.7 points. Gradient flow through multiple frozen model boundaries is empirically verified to be tractable, and the output node develops selective routing behavior across layer-2 nodes without explicit supervision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。