用AI工具自动分类社交应用用户评论中的隐私问题，帮开发者高效改进产品。
SENSOR: An ML-Enhanced Online Annotation Tool to Uncover Privacy Concerns from User Reviews in Social-Media Applications
- 基于GRU+CBOW+注意力机制的GRACE模型，自动标注评论类别。
- 在1.6万条评论上达到95.1%准确率，F1-score达0.9434。
- 适合关注用户隐私与产品迭代的开发团队使用。
社交媒体应用的普及引发了显著的隐私担忧，这些担忧常出现在用户评论中。评论不仅反映问题，也为开发者优化功能提供宝贵反馈。然而，评论数量庞大且表达复杂，手动识别和优先处理隐私相关问题对开发者而言极具挑战。现有研究虽已采用机器学习自动分类评论为隐私相关、无关、缺陷报告或功能请求等，但缺乏对隐私相关功能请求与缺陷报告的细粒度区分。本文提出SENtinel SORt（SENSOR），一个自动化在线标注工具，专用于将用户评论分类至上述三类。为此，提出GRACE模型（基于门控循环单元与连续词袋嵌入的注意力机制）。分析了来自Google Play Store的七款热门社交应用（包括Instagram、Facebook、WhatsApp、Snapchat、X、Facebook Lite和Line）的约1.6万条评论。两名标注者人工标注，获Cohen's Kappa值0.87，确保训练数据高一致性。在各类模型中，GRACE表现最佳：宏平均F1-score为0.9434，宏平均ROC-AUC为0.9934，准确率为95.10%，即使面对类别不平衡问题。SENSOR展现出显著潜力，可辅助开发者从用户评论中提取并应对隐私相关的功能建议或缺陷报告，从而提升用户隐私保护与信任。
原文摘要 · Abstract (English)
The widespread use of social media applications has raised significant privacy concerns, often highlighted in user reviews. These reviews also provide developers with valuable insights into improving apps by addressing issues and introducing better features. However, the sheer volume and nuanced nature of reviews make manual identification and prioritization of privacy-related concerns challenging for developers. Previous studies have developed software utilities to automatically classify user reviews as privacy-relevant, privacy-irrelevant, bug reports, feature requests, etc., using machine learning. Notably, there is a lack of focus on classifying reviews specifically as privacy-related feature requests, privacy-related bug reports, or privacy-irrelevant. This paper introduces SENtinel SORt (SENSOR), an automated online annotation tool designed to help developers annotate and classify user reviews into these categories. For automating the annotation of such reviews, this paper introduces the annotation model, GRACE (GRU-based Attention with CBOW Embedding), using Gated Recurrent Units (GRU) with Continuous Bag of Words (CBOW) and Attention mechanism. Approximately 16000 user reviews from seven popular social media apps on Google Play Store, including Instagram, Facebook, WhatsApp, Snapchat, X (formerly Twitter), Facebook Lite, and Line were analyzed. Two annotators manually labelled the reviews, achieving a Cohen's Kappa value of 0.87, ensuring a labeled dataset with high inter-rater agreement for training machine learning models. Among the models tested, GRACE demonstrated the best performance (macro F1-score: 0.9434, macro ROC-AUC: 0.9934, and accuracy: 95.10%) despite class imbalance. SENSOR demonstrates significant potential to assist developers with extracting and addressing privacy-related feature requests or bug reports from user reviews, enhancing user privacy and trust.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。