Lukas Blecher writes:
I've started to develop an Add-on for Blender 2.8x that can be useful for rotoscoping or masking objects in clips. The masking is done with the help of the neural network SiamMask which tracks the object and finds a mask for it in every frame. The mask quality is not perfect but can be used as a staring point for the artist. The amount of controll points can be regulated through a few parameters. The source code is hosted on GitHub.
20 Comments
Wow! That looks great. Will this be further developed?
There still are features missing in comparison to the marker tracking in Blender that I might add in later on but I can't say for sure.
If you're talking about the neural network performance/mask quality, that is an active topic of research in machine learning. Maybe I'll upgrade the Add-on with a better network in the future. But for now that's where the Add-on will stay.
What training data is used? Is recorded any where?
The network was trained on the VOT Dataset (http://www.votchallenge.net/). There are also weights available that were trained on the DAVIS dataset (https://davischallenge.org/). You may want to check out the repository of SiamMask (https://github.com/foolwood/SiamMask) for additional information
Excellent thanks for that, I want to have a look at them myself for another project I have in mind.
This is potentially very interesting. Also a line tracking (surface tracking of drawn line) tool would be very interesting for applying textures to 2D animation in the style of Netflix KLAUS.
Will this be free?
Yes, it's open source. You can just download it from GitHub: https://github.com/lukas-blecher/AutoMask
Will the final version be free? If so thanks a ton.
Also, how do I go about downloading and installing it? Just download the project root and, making sure it is zipped, import it into Blender like any other addon?
It's all here in the README: https://github.com/lukas-blecher/AutoMask
You have to install a few dependencies for python.
And yes all of this is free to download as of now
Simply awesome my friend!!
Love this concept. I look forward to it being developed further
WOW this is a freakn outstanding start.
I can't help but think you would be better off using Grease Pencil for this instead of bezier curves.
GP enables the shape's ability to change every frame offering a higher level of refinement, whereas bezier curves have always been clumsy.
The one thing that would be missing here would be the ability to feather the shapes for motion blur.
Feathering shapes could be possible through using the grease pencil sculpt tools by:
1.duplicating the grease pencil object (no a fan)
2.duplicating the layer and then sculpting the offset with a blur filter
3. Code it straight into grease using the above current code solutions
Bloody hell that is some seriously exciting work,
I will make a YouTube clip demonstrating what I wrote about for you. With your abilities, you will definitely be able to take it to the next level I'm sure of it.
Super exciting working for the industry. Amazing
Here is a video explaining what I was talking about:
https://www.youtube.com/watch?v=msgknlaMJgA&list=PL-zg5csjHY0goZ5b3puiJsPFXb15w8APo&index=2&t=0s
Thank you very much for the input. I didn't know about this option when I started the project. From what I understand it would have saved me some trouble but I won't go back and change it for now.
You might be right that the Bézier curves are computationally more demanding but they do approach the actual pixel mask by the neural network very nicely. Especially if the three controllable parameters are dialed down a bit. That results in more mask points and in return a better fit of the mask. For the demonstration I didn't want too many control points on screen.
I believe the bottleneck here is the neural network.
I hope I didn't miss your point
It will be interesting to see if someone picks up the torch and carries it from here. Thank for sharing.
Awsome!!!!
Can I ask about the changes that has to be done in the automask.py file? I'm having trouble about installing the add-on.
Same here. Did you find out how to do it?
Sorry guys, I'm not checking these comments regularly. If you have troubles please open an issue in the GitHub repo.
However, I'm currently quite busy but I'll see if I can help you.