arXiv:2608.16347cs.CLcs.LG2026-08

不同大模型间可因果传递激活状态,但效果依赖架构匹配。

Architecture-Dependent Causal Transfer of Activation States Across Large Language Models

  • 用学习投影网络在不同模型间转移激活状态,评估三重验证。
  • 仅一个模型对(Qwen2→Phi-3)实现显著输出相似性提升(23.3%)。
  • 结果表明状态迁移依赖架构,而非通用意义传递,适合跨模型研究者。

AI系统间的直接通信依赖自然语言作为中介,带来编码/解码开销、令牌成本和延迟。本文探讨是否可通过学习投影,在不同大语言模型(LLM)架构间因果性地传递内部激活状态。评估包含三个层次:表征相似性、投影状态的跨模型检索、生成过程中激活注入的端到端因果转移。使用四个架构差异显著的开源模型(Qwen2-0.5B、Phi-3-mini、Mistral-7B、FLAN-T5-base),发现训练后模型的表征对齐优于随机初始化基线,且基于秩的度量(互k近邻对齐)比中心核对齐(CKA)或Procrustes分析更鲁棒。对于三个解码器型模型对,学习投影网络在独立测试集上检索正确目标模型表征的准确率高达45–50%(随机为5%),而基于编码器的FLAN-T5表现仅达随机水平。在生成中注入投影激活,仅在一组解码器对(Qwen2-0.5B→Phi-3-mini)中观察到统计显著的因果效应(23.3%对比负控0.0%,p=0.047,FDR校正),另两组目标为Mistral-7B的对子虽有相似隐藏状态仍无效果。结果表明,仅表征载体可被因果转移,意义未被传递,当前方法具有架构依赖性。

原文摘要 · Abstract (English)

Direct communication between AI systems relies on natural language as an intermediate layer, incurring encoding/decoding overhead, token cost, and latency. We ask whether internal activation states can instead be transferred causally between different large language model (LLM) architectures via a learned projection, evaluated at three levels: representational similarity, cross-model retrieval from projected states, and end-to-end causal transfer via activation injection during generation. Using four architecturally diverse open-weight models (Qwen2-0.5B, Phi-3-mini, Mistral-7B, FLAN-T5-base), we find that representational alignment in trained models exceeds a random-initialization null baseline and is best captured by a rank-based metric (mutual k-nearest-neighbour alignment), more robust to activation-magnitude outliers than centered kernel alignment (CKA) or Procrustes analysis. A learned projection network retrieves the correct target-model representation from a held-out set well above chance for the three causal decoder-only model pairs (45-50% top-1 accuracy vs. 5% chance) but at chance level for the encoder-based FLAN-T5. Injecting projected activations into a target model during generation produces a statistically significant, pre-registered causal effect on retrieval-based output similarity for only one of the three decoder-only pairs (Qwen2-0.5B to Phi-3-mini: 23.3% vs. 0.0% under negative control, p=0.047, FDR-corrected); the two pairs targeting Mistral-7B show no such effect despite comparable representational alignment at the hidden-state level. We interpret these results as evidence for causal transfer of the representational vehicle, not of meaning, and conclude that end-to-end activation-state transfer between LLMs, as currently implemented, is architecture-dependent rather than universal.

大模型激活转移因果推断架构依赖

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。