Markerless face motion capture from a video file

garagefarm.net writes:

How to turn your video into facial motion capture data automatically using no markers at all? In this video DJ tests the new feature introduced by Deepmotion in their Animate 3d cloud service.

Creating a simple alien abduction scene from raw video recording, DJ shows how you can create a 3d avatar straight on Deepmotion's webpage (that can even use a photo for creating a likeness of your 3d character). With such a custom 3d character or a dummy model, you can easily create a full-rigged animation that you can download and use in Blender or any other 3d software like 3dsMax, Cinema 4D, or Maya. The facial animations are made using shape keys so if you'd like to use your own character with the service you need to make sure it’s properly set up with a rig and appropriate shape keys for the face features.

This new exciting AI technology still has a lot of flaws and is far from perfect, but with some small tweaks, it is already quite usable. DJ shows how you can simplify the animation curves to reduce the number of keyframes. This makes tweaking the animations a lot easier. Finding places that feel off in the animations is quite easy - after all, you have the reference video footage from the start. Compare it and tweak the curves at the right keyframes and with a little patience and hard work you can polish the animations to look presentable.

After all, creating them by hand from scratch would take much more time.

With the vTubers, metaverse, and other trends of that kind - this can be a real lifesaver!