针对大模型文本分类，提出分层嵌入融合方法，提升效果且无需微调。
Layer-Aware Embedding Fusion for LLMs in Text Classifications
- 按任务需求选择关键层级的嵌入表示，避免盲目使用全部层。
- 多模型嵌入融合可提升分类准确率，前提是模型具备互补性。
- 实验验证了不同数据集对应最优层不同，兼顾性能与资源开销。
嵌入融合已成为提升多种自然语言处理任务性能的有效方法，但如何选择最优层及设计有效融合策略仍缺乏系统指导。本文提出一种分层感知的嵌入选择方法，通过定量评估不同层在下游任务中的重要性，发现关键层随数据集变化而异。同时探索了不需模型微调即可结合多个大模型嵌入的融合方式，实验证明其在四个英文文本分类数据集（SST-2、MR、R8、R52）上能有效提升性能，前提是各模型具备互补表征能力。此外，研究还分析了融合带来的内存占用和推理时间开销，为实际部署提供平衡视角。未来工作将拓展至多语言与领域特定数据集，并探索自动化层选择技术以提升性能与可扩展性。
原文摘要 · Abstract (English)
Embedding fusion has emerged as an effective approach for enhancing performance across various NLP tasks. However, systematic guidelines for selecting optimal layers and developing effective fusion strategies for the integration of LLMs remain underexplored. In this study, we propose a layer-aware embedding selection method and investigate how to quantitatively evaluate different layers to identify the most important ones for downstream NLP tasks, showing that the critical layers vary depending on the dataset. We also explore how combining embeddings from multiple LLMs, without requiring model fine-tuning, can improve performance. Experiments on four English text classification datasets (SST-2, MR, R8, and R52) demonstrate that different layers in LLMs exhibit varying degrees of representational strength for classification, and that combining embeddings from different models can enhance performance if the models exhibit complementary characteristics. Additionally, we discuss resources overhead (memory and inference time) to provide a balanced perspective on the real world feasibility of embedding fusion. Future work will explore multilingual and domain specific datasets, as well as techniques for automating layer selection, to improve both performance and scalability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。