用大模型从芬兰难民口述史中提取社交关系,验证了开源模型的可用性。
Extracting Social Connections from Finnish Karelian Refugee Interviews Using LLMs
- 用零样本提示法从8.9万份访谈中提取家庭成员的社交组织与兴趣爱好
- GPT-4达到88.8%的F-score,接近人工水平;开源模型Llama-3-70B-Instruct达87.7%
- 用GPT-4生成数据微调FinBERT,在6千至3万样本下获84.1%~86.3%准确率
我们在89,339份战后从芬兰东卡累利阿迁居的难民家庭的芬兰语访谈文本上开展零样本信息抽取研究。目标有二:一是分别提取每位家庭成员的社交组织与爱好,作为其社会融入程度的代理变量;二是评估多种方法的适用性,比较多个生成式模型与监督学习方法的优劣。结果显示,最佳生成模型GPT-4的F-score为88.8%,接近人工水平;最佳开源模型Llama-3-70B-Instruct达到87.7%。此外,我们采用GPT-4生成训练数据微调芬兰BERT(FinBERT),仅用6,000份访谈即达84.1%的F-score,增至30,000份时提升至86.3%。该方法在资源受限或数据量大时更具吸引力。
原文摘要 · Abstract (English)
We performed a zero-shot information extraction study on a historical collection of 89,339 brief Finnish-language interviews of refugee families relocated post-WWII from Finnish Eastern Karelia. Our research objective is two-fold. First, we aim to extract social organizations and hobbies from the free text of the interviews, separately for each family member. These can act as a proxy variable indicating the degree of social integration of refugees in their new environment. Second, we aim to evaluate several alternative ways to approach this task, comparing a number of generative models and a supervised learning approach, to gain a broader insight into the relative merits of these different approaches and their applicability in similar studies. We find that the best generative model (GPT-4) is roughly on par with human performance, at an F-score of 88.8%. Interestingly, the best open generative model (Llama-3-70B-Instruct) reaches almost the same performance, at 87.7% F-score, demonstrating that open models are becoming a viable alternative for some practical tasks even on non-English data. Additionally, we test a supervised learning alternative, where we fine-tune a Finnish BERT model (FinBERT) using GPT-4 generated training data. By this method, we achieved an F-score of 84.1% already with 6K interviews up to an F-score of 86.3% with 30k interviews. Such an approach would be particularly appealing in cases where the computational resources are limited, or there is a substantial mass of data to process.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。