arXiv:2410.14715cs.CVcs.AI2024-10被引 2

用AI让远古三叶虫动起来,生成逼真视频还原其生活场景。

Animating the Past: Reconstruct Trilobite via Video Generation

  • 通过大模型自动生成视频提示词,结合真实化石数据优化生成效果。
  • 在9088张三叶虫化石图像基础上,生成视频视觉真实感显著优于现有方法。
  • 适合古生物学研究与科普教育,助力科学可视化创新。

古生物学依赖化石重建远古生态系统与演化过程。三叶虫作为重要的已灭绝海洋节肢动物,其保存完好的化石记录为理解古生代环境提供了关键信息。从静态化石重构三叶虫行为,将推动科学与教育领域的动态复原新标准。尽管文本到视频(T2V)技术具有潜力,但存在视觉真实性和连贯性不足等挑战,限制了其在科学场景中的应用。为此,我们提出一种自动化的T2V提示词学习方法:利用大语言模型生成微调视频生成模型的提示词,并通过量化视频视觉真实性和流畅性的奖励信号进行训练。视频模型微调与奖励计算基于包含9,088张Eoredlichia intermedia化石图像的数据集,该物种代表了三叶虫类群的典型视觉特征。定性与定量实验表明,本方法生成的三叶虫视频在视觉真实性上显著优于强大基线模型,有望提升科学认知与公众参与度。

原文摘要 · Abstract (English)

Paleontology, the study of past life, fundamentally relies on fossils to reconstruct ancient ecosystems and understand evolutionary dynamics. Trilobites, as an important group of extinct marine arthropods, offer valuable insights into Paleozoic environments through their well-preserved fossil records. Reconstructing trilobite behaviour from static fossils will set new standards for dynamic reconstructions in scientific research and education. Despite the potential, current computational methods for this purpose like text-to-video (T2V) face significant challenges, such as maintaining visual realism and consistency, which hinder their application in science contexts. To overcome these obstacles, we introduce an automatic T2V prompt learning method. Within this framework, prompts for a fine-tuned video generation model are generated by a large language model, which is trained using rewards that quantify the visual realism and smoothness of the generated video. The fine-tuning of the video generation model, along with the reward calculations make use of a collected dataset of 9,088 Eoredlichia intermedia fossil images, which provides a common representative of visual details of all class of trilobites. Qualitative and quantitative experiments show that our method can generate trilobite videos with significantly higher visual realism compared to powerful baselines, promising to boost both scientific understanding and public engagement.

古生物视频生成AI复原

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。