提出动态注意力调制机制，让大模型高效适应多任务且不遗忘旧知识。
Contextual Attention Modulation: Towards Efficient Multi-Task Adaptation in Large Language Models
- 通过上下文感知的注意力调制，动态增强任务特异性特征。
- 在多个任务上平均性能提升3.65%，优于现有方法。
- 适合需要多任务快速适配的大模型应用开发者。
大型语言模型具备强大的泛化能力，但在多任务适配中难以平衡知识保留与任务专精。传统微调易导致灾难性遗忘且资源消耗大，现有参数高效方法在复杂多任务场景下表现不佳。为此，我们提出上下文注意力调制（CAM），动态调节大模型自注意力模块的表征，增强任务特异性特征的同时保留通用知识，从而实现更高效、有效的多任务适应。在混合上下文注意力调制（HyCAM）框架中，结合一个全参数共享的CAM模块与多个轻量级专用CAM模块，并采用动态路由策略实现自适应知识融合。在问答、代码生成和逻辑推理等异构任务上的大量实验表明，该方法显著优于现有方案，平均性能提升3.65%。代码与数据已开源，便于复现。
原文摘要 · Abstract (English)
Large Language Models (LLMs) possess remarkable generalization capabilities but struggle with multi-task adaptation, particularly in balancing knowledge retention with task-specific specialization. Conventional fine-tuning methods suffer from catastrophic forgetting and substantial resource consumption, while existing parameter-efficient methods perform suboptimally in complex multi-task scenarios. To address this, we propose Contextual Attention Modulation (CAM), a novel mechanism that dynamically modulates the representations of self-attention modules in LLMs. CAM enhances task-specific features while preserving general knowledge, thereby facilitating more effective and efficient adaptation. For effective multi-task adaptation, CAM is integrated into our Hybrid Contextual Attention Modulation (HyCAM) framework, which combines a shared, full-parameter CAM module with multiple specialized, lightweight CAM modules, enhanced by a dynamic routing strategy for adaptive knowledge fusion. Extensive experiments on heterogeneous tasks, including question answering, code generation, and logical reasoning, demonstrate that our approach significantly outperforms existing approaches, achieving an average performance improvement of 3.65%. The implemented code and data are available to ease reproducibility at https://github.com/Applied-Machine-Learning-Lab/HyCAM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。