冻结解码器可加速训练并减少遗忘，适配多任务与多语言场景。
A Multi-Encoder Frozen-Decoder Approach for Fine-Tuning Large Language Models
- 采用多编码器+冻结解码器结构，降低部署开销。
- 在自然语言生成任务中表现优异，多语言任务遗忘率显著下降。
- 大模型搭配冻结解码器仍保性能，适合结构化与问答任务。
在参数高效微调方法中，冻结策略因能加速训练、减少灾难性遗忘并提升下游性能而广受关注。本文研究在包含多种自然语言任务的多任务设置下，冻结解码器的影响，旨在降低部署开销并增强对新任务的可移植性。基于Alexa™模型的实验表明，冻结解码器在具有自然语言输出的任务中表现良好，且能有效缓解多语言任务中的遗忘问题。此外，将冻结解码器与更大模型结合，可在结构化任务和问答任务中维持甚至提升性能，证明该策略适用于更广泛的任务类型。
原文摘要 · Abstract (English)
Among parameter-efficient fine-tuning methods, freezing has emerged as a popular strategy for speeding up training, reducing catastrophic forgetting, and improving downstream performance. We investigate the impact of freezing the decoder in a multi-task setup comprising diverse natural language tasks, aiming to reduce deployment overhead and enhance portability to novel tasks. Our experiments, conducted by fine-tuning both individual and multi-task setups on the AlexaTM model, reveal that freezing decoders is highly effective for tasks with natural language outputs and mitigates catastrophic forgetting in multilingual tasks. However, we find that pairing frozen decoders with a larger model can effectively maintain or even enhance performance in structured and QA tasks, making it a viable strategy for a broader range of task types.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。