用大模型生成合规文件，发现规则越模糊越难保证质量
From Regulation to Implementation: A Critical Evaluation of LLM-Assisted Regulatory Compliance in Industry
- 用大模型生成欧盟环保与隐私合规文档，测试不同规则下输出效果
- 模糊规则需更长提示词才能保证内容完整，严格规则则输出一致但易编造信息
- 适合关注AI合规应用落地的产业从业者和政策研究者参考
欧盟在可持续性与隐私监管方面处于领先地位，新法规普遍要求提供文档以确保合规。例如，生态设计产品法规（ESPR）引入数字产品护照（DPP）以实现全生命周期透明，而通用数据保护条例（GDPR）则要求进行数据保护影响评估（DPIA）以降低隐私风险。然而，生成这些合规文档极具挑战：工业数据通常格式多样、分散于企业及供应商系统中，难以提取并转化为符合DPP标准的格式；同时，DPIA需跨领域专业知识且无统一模板，对新系统开发极为困难。尽管已有研究提出使用大语言模型（LLM）辅助生成，但现有工作未充分评估数据提取指令与法规模糊性对输出质量的影响。本文通过对比不同模型在人工标注基准方案下的表现，发现对模糊格式如DPIA，需更长上下文提示以维持一致性与完整性；而对严格格式如数字电池护照（DBP），即使提示较短也能保持输出一致，但可能产生更多幻觉。
原文摘要 · Abstract (English)
The European Union (EU) has emerged as a leading regulatory body in the development of sustainability and privacy regulations. While new regulation requirements vary, many include a documentation artifact to ensure compliance. Notably, the Ecodesign for Sustainable Products Regulation (ESPR) introduces Digital Product Passports (DPPs) for life cycle transparency, while the General Data Protection Regulation (GDPR) mandates Data Protection Impact Assessments (DPIAs) to mitigate privacy risks. Creating these compliance artifacts, however, is challenging. Industrial data, which often exists in heterogeneous formats and is scattered across company and supplier systems, is required for DPPs and can be difficult to extract into compliant DPP formatting. Furthermore, DPIA documents require interdisciplinary expertise and follow no standardized format, making development difficult for novel systems. To address the particular complexity of compliance artifact creation for both regulations, researchers have proposed the use of LLMs in the generation process; however, the impact of the aforementioned problems on the output of these systems is largely unaddressed. This work investigates the existing research gap by exploring how data extraction instructions and regulatory vagueness impact the quality and consistency of LLM-produced compliance artifacts. The resulting artifacts are evaluated by benchmarking different models against manually created ground-truth schemas. The results reveal that less strict guidelines, such as DPIA formatting, require higher context prompts to maintain consistency and completeness. Stricter guidelines, such as formatting for Digital Battery Passports (DBP), result in consistent results regardless of prompt context, but may lead to more hallucinations in the output
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。