评估主权大模型的文化适配性与技术安全性
Assessing Socio-Cultural Alignment and Technical Safety of Sovereign LLMs
- 构建新数据集与分析框架，量化评估模型的文化契合度
- 发现主权模型虽支持低资源语言，但实际适配效果存疑
- 强调需综合安全与文化因素，避免盲目信任其用户适配性
当前大模型发展呈现对主权大模型日益增长的关注。全球关于主权大模型的讨论凸显各国需基于自身社会文化与历史背景开发本地化模型。然而，现有缺乏验证两大核心问题的框架与数据集：一是模型与用户社会文化背景的契合程度，二是其在不暴露用户风险的前提下是否具备技术稳健性与安全性。为此，我们构建了一个新数据集，并提出一套分析框架，用于提取与评估主权大模型的社会文化特征，同时评估其技术鲁棒性。实验结果表明，尽管主权大模型在支持低资源语言方面具有意义，但并未始终满足‘有效服务目标用户’的普遍主张。我们还发现，盲目接受该主张可能导致对安全等关键质量属性的低估。研究建议，推进主权大模型需采用更全面的评估体系，涵盖更广泛、扎实且实用的评价标准。
原文摘要 · Abstract (English)
Recent trends in LLMs development clearly show growing interest in the use and application of sovereign LLMs. The global debate over sovereign LLMs highlights the need for governments to develop their LLMs, tailored to their unique socio-cultural and historical contexts. However, there remains a shortage of frameworks and datasets to verify two critical questions: (1) how well these models align with users' socio-cultural backgrounds, and (2) whether they maintain safety and technical robustness without exposing users to potential harms and risks. To address this gap, we construct a new dataset and introduce an analytic framework for extracting and evaluating the socio-cultural elements of sovereign LLMs, alongside assessments of their technical robustness. Our experimental results demonstrate that while sovereign LLMs play a meaningful role in supporting low-resource languages, they do not always meet the popular claim that these models serve their target users well. We also show that pursuing this untested claim may lead to underestimating critical quality attributes such as safety. Our study suggests that advancing sovereign LLMs requires a more extensive evaluation that incorporates a broader range of well-grounded and practical criteria.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。