FizzleDorf's Animation Guide - Deforum

return to main page:
https://rentry.org/AnimAnon

Deforum

Introduction

This Page is an overview of the features and settings in the Deforum extension for the Automatic1111 Webui. If you are using the notebook in Google Colab, use this guide for the overview of controls (This is also a good alternate reference for A1111 users as well).

The Extension has a separate tab in the Webui after you install and restart. In it, there is another set of tabs for each section of Deforum. I'll be giving a breakdown of the controls, parameters and uses.

https://github.com/deforum-art/deforum-for-automatic1111-webui/wiki/Animation-Video-Examples-Gallery

Run

Contains most of the settings you should be familiar with in txt2img and img2img with some additions:

Keyframes

Contains parameters that can be keyframed in the animation sequence. The parameters are explained later in this guide. What is important to understand is the syntax for keyframing the parameters moving forward:

1
2
3

/*Abstracted Example
from frame 0 to frame 12, interpolate x to y.*/
0:(x), 12:(y)

"0:" and "12:" = keyframes to activate value
"(x)" and "(y)" = parameter values (pixels, CFG scale, Denoise strength, etc)

You must have "()" around the parameter value.

Values can be math functions. See below example:
//from the default zoom function to an altered function on frame 12.

1

0:(1.02+0.02*sin(2*3.14*t/20)), 12:(0.5+0.02*sin(2*3.14*t/20))

//The parameter values will interpolate between each other. If you want a value to be constant until a specific keyframe:

1

0:(x), 6:(x), 12: (y)

//from frame 0 to 6, maintain the same value. From frame 6 to 12, interpolate the values.

See Animation Settings for more details on individual parameters

Main Settings

Contains the animation mode selection, max frames for the animation sequence, and the border behavior.

Prompts

This is where prompts are keyframed in the animation sequence. The syntax is as follows:

1
2
3
4
5

//Abstracted Example
{
    "0": "Prompt A --neg NegPompt"
    "12": "Prompt B" 
}

where "0": and "12": = keyframes where prompt is resolved in interpolation.
Prompt A and B are positive prompts and NegPrompt is a negative prompt.
Positive prompts are written first
The negative prompt is read after --neg
The quotations must be surrounding the keyframe num and another set around the full prompt with the negative.

1
2
3
4
5
6
7

//Default example, notice the function syntax ' ' inside ( ) of the first prompt
{
    "0": "(scenic countryside:1.0), (cherry:`where(cos(6.28*t/10)>0, 1.8*cos(6.28*t/10), 0.001)`), (strawberry:`where(cos(6.28*t/10)<0, -1.8*cos(6.28*t/10), 0.001)`), snow, detailed painting by greg rutkowski --neg (cherry:`where(cos(6.28*t/10)<0, -1.8*cos(6.28*t/10), 0.001)`), (strawberry:`where(cos(6.28*t/10)>0, 1.8*cos(6.28*t/10), 0.001)`)",
    "60": "a beautiful (((banana))), trending on Artstation",
    "80": "a beautiful coconut --neg photo, realistic",
    "100": "a beautiful durian, trending on Artstation"
}

This is keyframed as a 100 frame animation but can go longer off the last prompt.

Prompt Constants

These prompt inputs use the same syntax as you would prompt normally in imagen. These inputs are prepended to of your keyframed prompts. This is very handy for keeping the same subject across all the frames while keeping the animation prompt clean and easy to read.

Composable Mask scheduling

Masks that are swapped over designated keyframes. You can keyframe regular masks or noise masks.

Init

Settings for initializing the animation from an image as well as mask settings. Pretty straight forward for setting up as it is very similar to img2img settings. The input image and mask can be set from a URL as seen in the default values.

Init Image

Mask Init

Video Init

Parseq

The Parseq dropdown is for parsing the JSON export from Parseq. I have a separate guide on how to use Parseq here.

Video Input

Video Output

Output Settings

Manual Settings

Frame Interpolation (RIFE)

Use RIFE and other Video Frame Interpolation methods to smooth out, slow-mo (or both) your output videos.

Animation Modes:

A drop-down of the available animation modes. There are four different types to choose from: Interpolation, 2D, 3D, and Video Input. Below is a brief description, examples and settings of each mode.

Note	Example
Before Moving on, It's important to understand how the x,y,z axis are used in film. the X-axis is horizontal, Y-axis is vertical, and the Z-axis is depth (distance from the camera to the canvas).

Interpolation Mode

This mode is purely for txt2img prompt interpolation using the AND operator. A more in depth explanation and techniques can be found here

In this section I will be going over carrying over techniques from my guide on prompt interpolation. I recommend reading through that first before continuing through this section to get the most out of it.

>>>>>Setup<<<<<

In order for things to work properly a few settings need to be changed (Assuming you are starting with default Deforum settings):

• The seed_behavior (under the "Run" tab) needs to be set to "fixed" (You can also use schedule to add seed interpolations, just make sure it isn't changing all the time)
• The animation_mode under the "Keyframes" tab needs to be set to "Interpolation".
• Color correction is disabled by default in this mode.

1
2

//Abstracted Example:
"0": "Prompt A: `1-t/(x)` AND Prompt B: `t/(x)`  --neg NegPrompt"

Where t = the current keyframe
x = the difference between keyframes.
Prompt A: 1-t/(x) starts with a weight of 1 and interpolates to 0.
Prompt B: t/(x) starts with a weight of 0 and interpolates to 1.

This is creating our own prompt interpolation as a sort of override.

Example:

Chaining and Looping Interpolation Clips Placeholder fix:

1
2
3
4
5
6
7

"prompts": {
        "0": "Masterpiece, best quality, 1girl, long grey hair, swept bangs, grey eyes, sketch, traditional art, (smiling:0.0): `1-(t-0)/(10)` AND Masterpiece, best quality, 1girl, long grey hair, swept bangs, grey eyes, sketch, traditional art, (smiling:0.2): `(t-0)/(10)`  --neg lowres, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry, monocolor",
        "10": "Masterpiece, best quality, 1girl, long grey hair, swept bangs, grey eyes, sketch, traditional art, (smiling:0.2): `1-(t-10)/(10` AND Masterpiece, best quality, 1girl, long grey hair, swept bangs, grey eyes, sketch, traditional art, (laughing:1.1): `(t-10)/(10)`  --neg lowres, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry, monocolor",
        "20": "Masterpiece, best quality, 1girl, long grey hair, swept bangs, grey eyes, sketch, traditional art, (laughing:1.2): `1-(t-20)/(10)` AND Masterpiece, best quality, 1girl, long grey hair, swept bangs, grey eyes, sketch, traditional art, (smiling:0.8): `(t-20)/(10)`  --neg lowres, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry, monocolor",
        "30": "Masterpiece, best quality, 1girl, long grey hair, swept bangs, grey eyes, sketch, traditional art, (smiling:0.8): `1-(t-30)/(10)` AND Masterpiece, best quality, 1girl, long grey hair, swept bangs, grey eyes, sketch, traditional art, (smiling:0.2): `(t-30)/(10)`  --neg lowres, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry, monocolor",
        "40": "Masterpiece, best quality, 1girl, long grey hair, swept bangs, grey eyes, sketch, traditional art, (smiling:0.2):`1-(t-40)/(10)` AND Masterpiece, best quality, 1girl, long grey hair, swept bangs, grey eyes, sketch, traditional art, (smiling:0.0):`(t-40)/(10)`  --neg lowres, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry, monocolor"
    }

>>>>>Storyboard<<<<<

Instead of using an X/Y grid, you can use the animation prompts to storyboard your sequence. I would recommend using the webui images browser extension for quick access to the storyboard sequence to check your key poses or transitions. You can find the extension here if you don't already have it: https://github.com/yfszzx/stable-diffusion-webui-images-browser

To do this, set up your prompts so they are 1 frame apart from each other. Also set the # of frames in the animation to the total number of prompts you have.

1
2
3
4
5
6
7

{
    "0": "Masterpiece, best quality, 1girl, long grey hair, swept bangs, grey eyes, sketch, traditional art, (smiling:0.0",
    "1": "Masterpiece, best quality, 1girl, long grey hair, swept bangs, grey eyes, sketch, traditional art, (smiling:0.2)",
    "2": "Masterpiece, best quality, 1girl, long grey hair, swept bangs, grey eyes, sketch, traditional art, (laughing:1.2)",
    "3": "Masterpiece, best quality, 1girl, long grey hair, swept bangs, grey eyes, sketch, traditional art, (smiling:0.8)",
    "4": "Masterpiece, best quality, 1girl, long grey hair, swept bangs, grey eyes, sketch, traditional art, (smiling:0.0)"
}

Using the image browser, review the storyboard to see if any prompts need changing without having to generate the whole sequence. The Keyed frames will be replicated when you change the keyframe values.

Guided Images

Guided images is somewhat self explanatory. Use images from a URL or PATH to guide the latent to specified keyframes. You can use this as a guided image tool or as a looper depending on your settings in the keyframe images field.

Default Value for keyframe guidance:

1
2
3
4
5
6
7

{
    "0": "https://user-images.githubusercontent.com/121192995/215279228-1673df8a-f919-4380-b04c-19379b2041ff.png",
    "50": "https://user-images.githubusercontent.com/121192995/215279281-7989fd6f-4b9b-4d90-9887-b7960edd59f8.png",
    "100": "https://user-images.githubusercontent.com/121192995/215279284-afc14543-d220-4142-bbf4-503776ca2b8b.png",
    "150": "https://user-images.githubusercontent.com/121192995/215279286-23378635-85b3-4457-b248-23e62c048049.jpg",
    "200": "https://user-images.githubusercontent.com/121192995/215279228-1673df8a-f919-4380-b04c-19379b2041ff.png"
}

2D Motion

Shared parameters in 2D and 3D animation modes. Simulates 2D motion of the canvas.

>>>>>Motion Parameters<<<<<

>>>>>Examples<<<<<

Angle	Zoom	Translation X	Translation Y	Transform Center X	Transform Center Y
2DMode, Border: wrap, angle: 0:(10)	2D mode, Border: Replicate, zoom: 0:(0.985)	2D Mode, Border: wrap, translation_x: 0:(-10),11:(-10),12:(15)	translation_y: 0:(-5),6:(-2),12:(5)	Transform Center X = 1, Transform Center Y = 0	Transform Center X = 0, Transform Center Y = 1
Angle Comparison Example	2D mode, Border: Wrap, "zoom": "0:(0.985)"	Translation X Comparison Example	Translation Y Comparison Example	Transform Center X = 0, Transform Center Y = 0	Transform Center X = 0.5 (this is centered), Transform Center Y = 2
	Zoom Comparison Example

Perspective Flip

2D canvas transformations. These parameters only work in 2D mode (You can now use 3D mode with any and all 2D Motion Parameters!) are meant to simulate 3D canvas rotations and movements. This is mainly for low vram machines that can't generate 3D mode animations efficiently or at all but can be used for neat effects in conjunction with 3D mode

>>>>>Motion Parameters<<<<<

>>>>>Examples<<<<<

perspective_flip_theta	perspective_flip_phi	perspective_flip_gamma	perspective_flip_fv
perspective_flip_theta: 0:(0), 12:(12), 24:(0)	perspective_flip_phi: "0:(t%15)	perspective_flip_gamma: 0:(0),12:(5),24:(0)	perspective_flip_fv: 0:(53), 12:(100),24:(53)
perspective_flip_theta Comparison Example	perspective_flip_phi Comparison Example	perspective_flip_gamma Comparison Example

3D Motion, Depth & FOV

3D canvas transformations. These parameters only work in 3D mode.

Motion Parameters

Translation Z	Rotation 3D X	Rotation 3D Y	Rotation 3D Z
translation_z: 0:(5), 12:(-6)	rotation_3d_x: 0:(0.5)	rotation_3d_y: 0:(0.5)	rotation_3d_z: 0:(0.5)
	rotation_3d_x Comparison Example	rotation_3d_y Comparison Example	rotation_3d_z Comparison example

Depth Warping

Fov settings

Strength and CFG Schedule

Parameter	Description	Keyframe Example
Strength Schedule	Denoise strength of the img2img process. Low values adhere more closely to the input image while higher values are more "creative".	-----
CFG Scale Schedule	Leave one value at frame 0 for a constant scale, can be keyframed.	cfg_scale_schedule: 0: (1), 12: (12)

Seed Behavior, SubSeed and Seed Schedules

Seed Behavior

How the seed behaves over time (keyframes). There are four different Settings in the Auto1111 extension.

Subseed Schedule

>>>>>Examples<<<<<

The following is to illustrate the effects of resizing the seed. The original resolution of the generated images is 512x512. All examples also have seed schedule enabled and using expression (1+t) in the schedule

512x512 (original resolution)	768x768	1024x1024	2048x2048

Seed Schedule

Parameter	Description	Example
Seed Schedule	This setting only comes into effect when the seed_behavior setting is set to Schedule. This parameter can keyframe seeds and interpolates between them over time.	seed_schedule: 0: (4294967293),12: (2096896498)

a handy expression for mimicking the behavior of the iter seed mode is `"desired seed" + t

Step and Sampler Schedules

Checkpoint and Clip Skip Schedules

Noise Settings

You can select and edit the behavior of perilin noise using the settings here.

Coherence

Color Coherence : Contains 3 different CC options: LAB, HSV, RGB as well as the option to turn it off.

Parameter	Description	Example
Legacy colormatch	applies colormatch before adding noise. You need to have this turned on if you plan on using tile controlnet or undesired artifacts can occur.	------
Optical Flow Cadence	Optical flow estimation for your in-between (cadence) frames (cadence setting needs to be greater than 1 in order to have an effect)	-----
Optical Flow Generation	Generates each frame twice in order to capture the optical flow from the previous image to the first generation, then warps the previous image and redoes the generation	-----
Contrast Schedule	adjusts the overall contrast per frame (default neutral at 1.0)	contrast_schedule: 0: (1.0), 6: (0.5),12: (2)
color_force_grayscale	Converts the output video to grayscale.	-----

Anti Blur

ControlNet

Deforum also had controlnet implementation which is really handy for rotoscoping videos. A lot of the controls are the same save for the video and video mask inputs. You will notice a lot of flickering in the raw output. There is ways to mitigate this such as the Ebsynth utility, diffusion cadence (under the Keyframes Tab) or frame interpolation (Deforum has it's own implementation of RIFE. see Outputs section for details). Video Loopback is also another way to mitigate this (it will take trial and error to get the right settings).

To use this implementation, select enable to reveal the parameters. Init image (under the Init tab) also needs to be checked. You can use controlnet in any of the animation modes. I'll be going more in depth with each one in time. Below are parameter explanations that can help get the most out of controlnet in animations. Any other settings not covered can be learned about in This Rentry Guide.

Some tips for using controlnet in Deforum:

More info about ControlNets can be found in the Rotoscope guide

Hybrid Video Mode

Video mix settings for 2D and 3D modes using compositing methods and masking. A document was put together by the dev who implemented this. You can read about it here

Math Presets for Parameters

The Deforum math articleis a handy guide for understanding how to use math in Deforum . Desmos calculator is a handy tool for planning your oscillations and plots with a visualization as well. Supported Math functions can be found Here.

Sine	Cosine	abs(cos)	abs(sin)
D+A*sin(23.14*t/P)-3)	D+A*cos(23.14*t/P)-3	(A-(abs(cos(10*t/P))*B))	(A-(abs(sin(10*t/"P"))*B))
10*sin(2*3.14*t/10)	10*cos(2*3.14*t/10)	0.6-(abs(cos(10*t/5+0))*0.2)	0.6-(abs(cos(10*t/5+0))*0.2)
Modulus	Linear	Triangle	Fourier
A*(t%5)+P	A*t+D	(2 + 2("A"))/3.14*arcsin(sin((2*3.14)/ "P" *t))	D + (A*(sin*t/P)+sin(A*t/P) + sin(A*t/P)
0.375*(t%5)+15	t	(2(1))/3.14*arcsin(sin((2*3.14)/1*t))	0 + (1*sin(t)-(sin(6*t)/0.1)+(sin(8*t)/0.1)) + (sin((12*t)/0.1))
Square	-----	-----	-----
D+A*0**0**(0-sin(3.14*t/P))	-----	-----	-----
5*0**0**(0-sin(3.14*t/2))	-----	-----	-----
	-----	-----	-----

Handy Resources

return to main page:
https://rentry.org/AnimAnon