用音乐生成个性化视频，还能保护人脸隐私。
Secure & Personalized Music-to-Video Generation via CHARCHA
- 结合歌词、节奏与情感生成定制画面。
- 通过低秩适配技术让视频贴合用户形象。
- 引入身份验证协议，防止人脸滥用。
音乐是高度个人化的体验，本文旨在通过全自动流程实现个性化音乐视频生成。该方法使听众不仅是观众，更成为创作者，基于音乐的歌词、节奏和情绪生成个性化、一致且情境相关的视觉内容。系统融合多模态翻译与生成技术，并利用低秩适应（low-rank adaptation）对用户图像进行处理，生成反映音乐特征与个体特征的沉浸式视频。为保障用户身份安全，本文提出CHARCHA（专利待审），一种面部身份验证协议，可在确保用户授权的前提下收集其图像用于个性化视频生成，有效防止未经授权的面部使用。本研究提供了一种安全且创新的深度个性化音乐视频生成框架。
原文摘要 · Abstract (English)
Music is a deeply personal experience and our aim is to enhance this with a fully-automated pipeline for personalized music video generation. Our work allows listeners to not just be consumers but co-creators in the music video generation process by creating personalized, consistent and context-driven visuals based on lyrics, rhythm and emotion in the music. The pipeline combines multimodal translation and generation techniques and utilizes low-rank adaptation on listeners' images to create immersive music videos that reflect both the music and the individual. To ensure the ethical use of users' identity, we also introduce CHARCHA (patent pending), a facial identity verification protocol that protects people against unauthorized use of their face while at the same time collecting authorized images from users for personalizing their videos. This paper thus provides a secure and innovative framework for creating deeply personalized music videos.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。