arXiv:2507.13236cs.CL2025-07被引 6

不改参数也不扩输入,用激活值引导实现跨任务知识迁移

Enhancing Cross-task Transfer of Large Language Models via Activation Steering

  • 通过分析隐藏层激活模式,发现不同任务间有可复用的增强特征
  • 在低资源任务上达到比基线更高的准确率,计算成本更低
  • 适合需要快速适配新任务且数据稀缺的场景

大语言模型虽能通过提示词利用预训练知识,但在未见任务尤其是数据稀缺场景下表现不佳。尽管跨任务上下文学习可直接实现知识迁移,仍面临鲁棒性、可扩展性和效率挑战。本文研究是否可通过不更新参数或扩展输入的方式,在隐空间中实现跨任务迁移。通过对模型隐空间激活模式的分析,我们发现上下文示例引发的增强激活在不同任务间具有稳定模式。受此启发,提出CAST框架——一种无需参数更新的跨任务激活引导迁移方法。该方法先从高资源任务中选取有影响力且多样化的样本,再利用其对比增强的激活表示,引导模型适应低资源任务。在跨领域与跨语言迁移设置下的大量实验表明,该方法优于现有基线,具备更优可扩展性与更低计算开销。

原文摘要 · Abstract (English)

Large language models (LLMs) have shown impressive abilities in leveraging pretrained knowledge through prompting, but they often struggle with unseen tasks, particularly in data-scarce scenarios. While cross-task in-context learning offers a direct solution for transferring knowledge across tasks, it still faces critical challenges in terms of robustness, scalability, and efficiency. In this paper, we investigate whether cross-task transfer can be achieved via latent space steering without parameter updates or input expansion. Through an analysis of activation patterns in the latent space of LLMs, we observe that the enhanced activations induced by in-context examples have consistent patterns across different tasks. Inspired by these findings, we propose CAST, a novel Cross-task Activation Steering Transfer framework that enables effective transfer by manipulating the model's internal activation states. Our approach first selects influential and diverse samples from high-resource tasks, then utilizes their contrastive representation-enhanced activations to adapt LLMs to low-resource tasks. Extensive experiments across both cross-domain and cross-lingual transfer settings show that our method outperforms competitive baselines and demonstrates superior scalability and lower computational costs.

大模型迁移激活引导零样本迁移

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。