用AI工具链把敏感文档转成可分析数据，保护隐私还能批量处理。
Transforming Sensitive Documents into Quantitative Data: An AI-Based Preprocessing Toolchain for Structured and Privacy-Conscious Analysis
- 用大模型标准化、摘要化并翻译文本，统一格式提升可分析性。
- 处理10,842份瑞典法庭文件，成功去标识化且保留语义内容。
- 全本地运行、开源模型，适合医疗法律等隐私敏感领域的研究。
来自法律、医疗和行政来源的非结构化文本为公共卫生与社会科学研究提供了丰富但未充分利用的资源。然而，大规模分析面临两大挑战：包含敏感个人身份信息，以及结构和语言的高度异质性。本文提出一个模块化工具链，将此类文本数据转化为基于嵌入的分析可用格式，仅依赖可在本地硬件上运行的开源模型，仅需工作站级GPU，支持隐私敏感型研究。该工具链利用大语言模型（LLM）提示技术对文本进行标准化、摘要化，并在需要时翻译为英文以增强可比性。去标识化通过基于LLM的删除技术实现，辅以命名实体识别与规则方法，最大限度降低泄露风险。我们在瑞典《施虐者照护法》（LVM）下的10,842份法庭判决文书（超过56,000页）上验证了该工具链，每份文档均被转化为匿名化、标准化的摘要，并生成文档级嵌入向量。通过人工审查、自动扫描及预测评估验证，结果表明该工具链能有效去除识别信息，同时保留语义内容。作为示范应用，我们使用少量人工标注摘要训练了一个预测模型，展示了该工具链在大规模半自动化内容分析中的潜力。该工具链使敏感文档的结构化、隐私保护分析成为可能，为因隐私与异质性限制而此前难以获取文本数据的研究领域开辟新路径。
原文摘要 · Abstract (English)
Unstructured text from legal, medical, and administrative sources offers a rich but underutilized resource for research in public health and the social sciences. However, large-scale analysis is hampered by two key challenges: the presence of sensitive, personally identifiable information, and significant heterogeneity in structure and language. We present a modular toolchain that prepares such text data for embedding-based analysis, relying entirely on open-weight models that run on local hardware, requiring only a workstation-level GPU and supporting privacy-sensitive research. The toolchain employs large language model (LLM) prompting to standardize, summarize, and, when needed, translate texts to English for greater comparability. Anonymization is achieved via LLM-based redaction, supplemented with named entity recognition and rule-based methods to minimize the risk of disclosure. We demonstrate the toolchain on a corpus of 10,842 Swedish court decisions under the Care of Abusers Act (LVM), comprising over 56,000 pages. Each document is processed into an anonymized, standardized summary and transformed into a document-level embedding. Validation, including manual review, automated scanning, and predictive evaluation shows the toolchain effectively removes identifying information while retaining semantic content. As an illustrative application, we train a predictive model using embedding vectors derived from a small set of manually labeled summaries, demonstrating the toolchain's capacity for semi-automated content analysis at scale. By enabling structured, privacy-conscious analysis of sensitive documents, our toolchain opens new possibilities for large-scale research in domains where textual data was previously inaccessible due to privacy and heterogeneity constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。