arXiv:2603.18822cs.CL2026-03被引 1

用多阶段框架在噪声俄语社交文本中识别人类基本价值观

Detecting Basic Values in A Noisy Russian Social Media Text Data: A Multi-Stage Classification Framework

  • 分阶段过滤垃圾与非个人内容,结合LLM标注生成软标签
  • 最佳模型XLM RoBERTa large在测试集上F1宏值达0.83
  • 发现开放性变革价值被系统性高估,适合社会文化研究者

本研究提出一种多阶段分类框架,用于在750万条公开俄语社交文本中检测人类价值观。基于舒瓦茨的基本价值观理论,构建包含垃圾内容过滤、价值相关文本筛选、LLM标注及多标签分类的流程。通过多位专家标注作为解释性基准而非绝对真值,采用多个LLM判断聚合生成反映共识程度的软标签,训练基于Transformer的模型预测十类基本价值观概率。最优模型XLM RoBERTa large在留出测试集上取得0.83的F1宏值和0.71的平均F1。结果表明,模型总体与人工判断一致,但系统性高估了开放性变革维度。研究揭示了俄语社交网络中价值观表达及其共现模式,推动数字环境中文化差异、话语建构与价值解读的跨学科研究。所有模型已公开发布。

原文摘要 · Abstract (English)

This study presents a multi-stage classification framework for detecting human values in noisy Russian language social media, validated on a random sample of 7.5 million public text posts. Drawing on Schwartz's theory of basic human values, we design a multi-stage pipeline that includes spam and nonpersonal content filtering, targeted selection of value relevant and politically relevant posts, LLM based annotation, and multi-label classification. Particular attention is given to verifying the quality of LLM annotations and model predictions against human experts. We treat human expert annotations not as ground truth but as an interpretative benchmark with its own uncertainty. To account for annotation subjectivity, we aggregate multiple LLM generated judgments into soft labels that reflect varying levels of agreement. These labels are then used to train transformer based models capable of predicting the probability of each of the ten basic values. The best performing model, XLM RoBERTa large, achieves an F1 macro of 0.83 and an F1 of 0.71 on held out test data. By treating value detection as a multi perspective interpretive task, where expert labels, GPT annotations, and model predictions represent coherent but not identical readings of the same texts, we show that the model generally aligns with human judgments but systematically overestimates the Openness to Change value domain. Empirically, the study reveals distinct patterns of value expression and their co-occurrence in Russian social networks, contributing to a broader research agenda on cultural variation, communicative framing, and value based interpretation in digital environments. All models are released publicly.

价值观检测多阶段框架俄语文本LLM标注

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。