You're blocking ads, which pay for BlenderNation. Read about other ways to support us.

AutoMask - Using Machine Learning for Video Masking

20

Lukas Blecher writes:

I've started to develop an Add-on for Blender 2.8x that can be useful for rotoscoping or masking objects in clips. The masking is done with the help of the neural network SiamMask which tracks the object and finds a mask for it in every frame. The mask quality is not perfect but can be used as a staring point for the artist. The amount of controll points can be regulated through a few parameters. The source code is hosted on GitHub.

20 Comments

    • There still are features missing in comparison to the marker tracking in Blender that I might add in later on but I can't say for sure.
      If you're talking about the neural network performance/mask quality, that is an active topic of research in machine learning. Maybe I'll upgrade the Add-on with a better network in the future. But for now that's where the Add-on will stay.

  1. Johan Edström on

    This is potentially very interesting. Also a line tracking (surface tracking of drawn line) tool would be very interesting for applying textures to 2D animation in the style of Netflix KLAUS.

  2. Will the final version be free? If so thanks a ton.

    Also, how do I go about downloading and installing it? Just download the project root and, making sure it is zipped, import it into Blender like any other addon?

  3. WOW this is a freakn outstanding start.

    I can't help but think you would be better off using Grease Pencil for this instead of bezier curves.

    GP enables the shape's ability to change every frame offering a higher level of refinement, whereas bezier curves have always been clumsy.

    The one thing that would be missing here would be the ability to feather the shapes for motion blur.

    Feathering shapes could be possible through using the grease pencil sculpt tools by:
    1.duplicating the grease pencil object (no a fan)
    2.duplicating the layer and then sculpting the offset with a blur filter
    3. Code it straight into grease using the above current code solutions

    Bloody hell that is some seriously exciting work,

    I will make a YouTube clip demonstrating what I wrote about for you. With your abilities, you will definitely be able to take it to the next level I'm sure of it.

    Super exciting working for the industry. Amazing

  4. Can I ask about the changes that has to be done in the automask.py file? I'm having trouble about installing the add-on.

      • Sorry guys, I'm not checking these comments regularly. If you have troubles please open an issue in the GitHub repo.
        However, I'm currently quite busy but I'll see if I can help you.

Leave A Reply

To add a profile picture to your message, register your email address with Gravatar.com. To protect your email address, create an account on BlenderNation and log in when posting a message.