Zeroscope text-to-videoA settings guide by Replicate
Zeroscope v2 is an open-source text-to-video model, give it a prompt and it’ll generate a short video.Make videos with Replicate
num_inference_steps to see what happens.
Start with the 576w model
Begin by generating videos using the
576w model at 576x320 resolution. When you find a video that you like, upscale it to 1024x576 using the
You could create a 1024x576 video directly by using the
xl model, but this video will render with duplicate objects and will tend to have low coherency. It’s better to use a two-step process (unless you’re intentionally trying to get weirder results).
Number of frames and FPS (frames-per-second)
Zeroscope v2 was trained on short 24 frame clips. Set your
num_frames to 24 for the best results. You can try longer clips, but beyond 40 frames the video will completely degrade.
For a smooth 1s video set the
fps to 24. Alternatively, use 12 or 8 for a more jerky 2s or 3s video, then use a video interpolater to fill in the missing frames and make the video smooth again.
This determines how much the model pays attention to your prompt. Too low and you see a grayscale mess, too high and the video will look distorted, with color artifacts. The sweet-spot is between 10 and 15, but try pushing this higher if your prompt is being ignored. Try setting
guidance_scale to 12.5 to start with.
How many inference steps will be used to generate your video. More steps give better quality and coherency, but takes longer to generate. Fewer steps will give a lower quality video but will generate quickly – a good setting for fast prompt experimentation. Going above 100 steps will not improve your video.
num_inference_steps to 50 to start with.
When you have your 576x320 video, you can upscale it with the
- reuse the same prompt and negative prompt
init_videoto the video you want to upscale
- pick an
init_weight, try 0.2
- use a 1024x576 resolution
Try experimenting with the prompt too, you can get some weird results.
Smooth out your video with frame interpolation.