Procedural Animation Generation Using LSTM-VAE

Procedurally generate FBX animations efficiently with small training sets!

Workflow

-> Add all FBX training animations into the input_animations directory.
-> Start Blender and run bpy_convert_input.py using the Blender Development VSCode extension.
This will import all the FBX animations into Blender, record the rotations of each bone and store this data into a text file to be used for training in the next step.
-> Run lstm_vae.py and adjust training parameters as needed.
Alter sample_outputs if you wish to change the number of generated animations outputted.
The generated animation data will be in the generated_animation_data directory.
-> Run bpy_convert_gen_data.py which will convert the generated data into FBX files.
Done! 🎉 The generated_fbx directory will hold all the generated animations and can be used in any projects, e.g. Unity etc.
The generated animations should automatically smoothly transition from frame to frame, but feel free to add any post-processing you would like!

Notes

• bpy_convert_input.py assumes all animations have the same named bones, in the same order. To deal with this, rename similar bones and ensure the bone order when written to and read from a file remain in the exact same order.
• You can use fake-bpy-module to access the Blender Python API for code completion within your IDE.
• Ensure that animation_max_frames in bpy_convert_input.py and bpy_convert_gen_data.py and seq_length in lstm_vae.py all have the same value.

Folder Structure

• character_rig contains the rigged model for the generated animations to be applied to.
• generated_animation_data contains the generated animation data as a series of rotations for each bone.
• generated_fbx contains the generated animations as FBX files which can be used in other software etc.
• input_animation_data contains the training animation data as a series of rotations.
• input_animations contains the set of FBX animations to be used to train the VAE.

Example Details

The example uses and trains the VAE on animations downloaded from Mixamo.
Mixamo Animation download settings:
-> Format: FBX Binary
-> Skin: Without Skin
-> Frames per Second: 30
-> Keyframe Reduction: None

The training dataset only contains 16 animations, yet still produces realistic animations.
The sitting and dancing animations only used 7 animations each, additionally.

Model Architecture

Encoder: 2 layered LSTM
Decoder: 2 layered LSTM
Mean and Var: 1 Fully Connected Layer
Optimiser: Adam