用简单任务的嵌入评估知识迁移,发现语义相似性任务最有助于指代消解。
Reverse Probing: Evaluating Knowledge Transfer via Finetuned Task Embeddings for Coreference Resolution
- 反向探查:用简单任务嵌入评估对复杂任务的迁移效果
- 语义相似性任务嵌入在指代消解中表现最佳,多任务融合提升性能
- 微调模型中间层嵌入优于最终层,注意力融合效果显著
本文重新思考经典探查方法,旨在评估从简单源任务到复杂目标任务的知识迁移。不同于传统在复杂源任务上探测多个简单目标任务的做法,本文探索多个简单源任务的嵌入对单一复杂目标任务(指代消解)的有效性。选择需要上下文理解的指代消解作为目标任务,测试了句法/语义简单任务(如复述检测、命名实体识别、关系抽取)的嵌入效果。通过系统实验,评估单个与组合任务嵌入的影响。结果表明,任务嵌入在指代消解中的效用差异显著,语义相似性任务(如复述检测)最具价值;微调模型中间层的表示常优于最终层;多任务嵌入结合可稳定提升性能,注意力聚合带来明显增益。这些发现揭示了任务特定表示与其适配复杂下游任务的关系,推动嵌入层面的任务迁移研究。
原文摘要 · Abstract (English)
In this work, we reimagine classical probing to evaluate knowledge transfer from simple source to more complex target tasks. Instead of probing frozen representations from a complex source task on diverse simple target probing tasks (as usually done in probing), we explore the effectiveness of embeddings from multiple simple source tasks on a single target task. We select coreference resolution, a linguistically complex problem requiring contextual understanding, as focus target task, and test the usefulness of embeddings from comparably simpler tasks tasks such as paraphrase detection, named entity recognition, and relation extraction. Through systematic experiments, we evaluate the impact of individual and combined task embeddings. Our findings reveal that task embeddings vary significantly in utility for coreference resolution, with semantic similarity tasks (e.g., paraphrase detection) proving most beneficial. Additionally, representations from intermediate layers of fine-tuned models often outperform those from final layers. Combining embeddings from multiple tasks consistently improves performance, with attention-based aggregation yielding substantial gains. These insights shed light on relationships between task-specific representations and their adaptability to complex downstream tasks, encouraging further exploration of embedding-level task transfer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。