用AI自动抓取网络文化数据,提升印尼文化遗产库的覆盖与深度。
Artificial Intelligence-Assisted Digital Inventory of Cultural Heritage & Traditional Knowledge: Case for Indonesian Open Digital Library of Culture
- 构建五阶段经济漏斗框架,实现多语言文化数据自动采集与整合。
- 通过贝叶斯证据融合与唯一发布机制,确保数据真实且不重复。
- 保留人类把关角色,兼顾自动化效率与文化敏感性,适合数字人文项目参考。
印尼文化数字图书馆(PDBI)自2007年起通过公众参与收集了数万条南洋文化遗产条目。手动贡献面临三大结构性障碍:覆盖不足(知识分散于多语言和多平台)、完整性差(公开源混杂真实与噪声信息)、完整度低(记录浅层)。本文提出一种基于AI的自主知识采集方法论框架,旨在扩大语料覆盖并深化每条记录的数据深度。该框架采用五阶段经济漏斗设计:定向爬取、多语言提取与标准化、向量编码与阻断、代理决策、幂等式发布,遵循确定性编排与代理决策原则。各阶段均形式化:漏斗经济学与最优过滤排序;爬取前沿动态作为亚临界分支过程,解释需反复重置种子的必要性;事实级新颖性通过包含度量衡量;贝叶斯多源证据融合,对神圣类别设置更高发布阈值;通过幂等插入与事务型消息队列实现精确一次效果;滑动窗口推理预算与预留协议;统计质量审计;种子选择建模为子模覆盖最大化。框架保留四项高价值人工角色:方向制定者、升级审批人、质量审计员、意义守门人,逐步提升机器自主性。伦理、法律与文化敏感性问题亦被讨论,架构上保证机器永不覆盖人类贡献。
原文摘要 · Abstract (English)
The Indonesian Digital Library of Culture (Perpustakaan Digital Budaya Indonesia, PDBI; budaya-indonesia.org) is a participatory platform that has collected tens of thousands of entries on Nusantara cultural heritage through public contribution since 2007. Manual contribution faces three structural barriers: coverage (knowledge is scattered across languages and sites), integrity (open sources mix authentic documentation with noise), and completeness (subjects are recorded but their data remain shallow). This paper presents a methodological framework for autonomous, AI-based harvesting of cultural knowledge from the open web, designed to expand corpus coverage while intensifying per-entry data depth. The methodology is organised as a five-stage economic funnel: focused crawling, multilingual extraction and canonicalisation, vector encoding with blocking, agentic decision-making, and idempotent publication, under the principle of deterministic orchestration, agentic decisions. Each stage is formalised: funnel economics and optimal filter ordering; crawl-frontier dynamics as a subcritical branching process that explains the necessity of recurrent re-seeding; fact-level novelty via a containment measure; Bayesian multi-source evidence fusion with elevated publication thresholds for sacred categories; exactly-once effects via idempotent upserts and the transactional outbox; sliding-window inference budgeting with a reservation protocol; statistical quality auditing; and seed selection as submodular coverage maximisation. The framework retains four high-value human roles: curator of direction, escalation approver, quality auditor, and guardian of meaning, while machine autonomy is raised in stages. Ethical, legal, and cultural-sensitivity implications are discussed, including the architectural guarantee that the machine never overwrites human contributions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。