.png){kind=icon}
返回
Still-Moving: Customized Video Generation without Customized Video Data
.png){kind=icon}
OA
AI
DOI：10.1145/3687945.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Customizing text-to-image (T2I) models has seen tremendous progress recently, particularly in areas such as personalization, stylization, and conditional generation. However, expanding this progress to video generation is still in its infancy, primarily due to the lack of customized video data. In this work, we introduce Still-Moving, a novel generic framework for customizing a text-to-video (T2V) model, without requiring any customized video data. The framework applies to the prominent T2V design where the video model is built over a T2I model (e.g., via inflation). We assume access to a customized version of the T2I model, trained only on still image data (e.g., using DreamBooth). Naively plugging in the weights of the customized T2I model into the T2V model often leads to significant artifacts or insufficient adherence to the customization data. To overcome this issue, we train lightweight Spatial Adapters that adjust the features produced by the injected T2I layers. Importantly, our adapters are trained on frozen videos (i.e., repeated images), constructed from image samples generated by the customized T2I model. This training is facilitated by a novel Motion Adapter module, which allows us to train on such static videos while preserving the motion prior of the video model. At test time, we remove the Motion Adapter modules and leave in only the trained Spatial Adapters. This restores the motion prior of the T2V model while adhering to the spatial prior of the customized T2I model. We demonstrate the effectiveness of our approach on diverse tasks including personalized, stylized, and conditional generation. In all evaluated scenarios, our method seamlessly integrates the spatial prior of the customized T2I model with a motion prior supplied by the T2V model.
Keyword:
Video Customization
Video Personalization
Video Stylization
Video Editing
Video Diffusion Models
期刊
IF:
9.5
论文数:
4.7K
被引数:
3.6W
机构
查看更多机构
引用论文
Sensorless Control of Z Source Inverter fed BLDC Motor Drive by FOC - DTC Hybrid Control Strategy Using Fuzzy Logic Controller采用模糊逻辑控制器的foc-dtc混合控制策略的Z源逆变器馈电BLDC电机驱动的无传感器控制
.png){kind=icon}
PREAI
分享收藏Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models基于编码器的域调整，用于文本到图像模型的快速个性化OAAI
OAAI分享收藏分享收藏
没有更多内容