Facial motion tracking

Allan Liddle shares how he did a facial motion capture project in Blender.

I made this video is an experiment for a potential customer: to prove whether I can successfully track the movements from a face in a video - as well as the expressions that go with it - and to project those movements onto the modelled face of another character with different facial proportions. It was quite a mission!

Here is the result (don't be too concerned about the accent ;-) :

https://youtu.be/3zg_7HCjUKI

Everything was done in Blender and a bit of Gimp. I downloaded the facial texture from the Internet, but I had to edit it in Gimp.

Below is the composite footage that compares the animation with the original video. Notice the differences in facial features, which had to be catered for.

Summary of how it was done:

I first built the model of the character's face, and then rigged it using Pitchypoy.

Next, I made a video of my face. I then used camera tacking to move the model of my face and to track the movements of my eyes, mouth, etc. Thereafter, I used the markers in the video to drive empties that run along the surface of the model of my face (as it moves).

There are bones in the rig of my face that then track those empties. The movements of those bones (in my face model) are then translated into movements of bones for the character's rig that I also developed. Some of the bones of the Pitchypoy rig of the front character then copy the location of the bones that are so moved.

(I hope it all makes sense)