首个中日双语检测自毁行为内容的基准,揭示文化比语言更影响模型效果。
JiraiBench: A Bilingual Benchmark for Evaluating Large Language Models' Detection of Human Self-Destructive Behavior Content in Jirai Community
- 构建中日双语自毁行为数据集,涵盖10419条中文、5000条日文帖子。
- 日文提示在中文内容上表现优于中文提示,跨文化迁移意外有效。
- 强调文化语境对多语言内容审核的关键作用,适合安全与伦理研究者。
本文提出JiraiBench,首个针对中日社交媒体中自毁行为内容检测的双语评估基准。聚焦跨国“Jirai”(地雷)亚文化,涵盖药物过量、进食障碍和自残等多重自毁行为,构建包含10,419条中文帖子和5,000条日文帖子的数据集,进行三类行为的多维标注,达成较高一致性。对四种先进模型的实验显示,指令语言显著影响性能:使用日文提示处理中文内容时反而优于中文提示,暗示文化接近性可能超越语言相似性。跨语言迁移实验表明,无需目标语言训练即可实现知识转移。研究强调需结合文化因素进行多语言内容监管,为构建更有效的在线弱势群体保护系统提供实证支持。
原文摘要 · Abstract (English)
This paper introduces JiraiBench, the first bilingual benchmark for evaluating large language models' effectiveness in detecting self-destructive content across Chinese and Japanese social media communities. Focusing on the transnational "Jirai" (landmine) online subculture that encompasses multiple forms of self-destructive behaviors including drug overdose, eating disorders, and self-harm, we present a comprehensive evaluation framework incorporating both linguistic and cultural dimensions. Our dataset comprises 10,419 Chinese posts and 5,000 Japanese posts with multidimensional annotation along three behavioral categories, achieving substantial inter-annotator agreement. Experimental evaluations across four state-of-the-art models reveal significant performance variations based on instructional language, with Japanese prompts unexpectedly outperforming Chinese prompts when processing Chinese content. This emergent cross-cultural transfer suggests that cultural proximity can sometimes outweigh linguistic similarity in detection tasks. Cross-lingual transfer experiments with fine-tuned models further demonstrate the potential for knowledge transfer between these language systems without explicit target language training. These findings highlight the need for culturally-informed approaches to multilingual content moderation and provide empirical evidence for the importance of cultural context in developing more effective detection systems for vulnerable online communities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。