Blender Live Shape – Real-Time Displacement Modifier
Sebastian Korczak likes to hook up gear to Blender. Check out his latest experiment: Blender Live Shape – a real-time displacement modifier (similar to normal map) based on live grayscale video from webcam. Read on for more of his work.
Sebastian writes:For years I’m interested in Blender but this year I started to make some electronic project connected with Blender. Probably you like to see my concept of real-time 3D object in Blender controlled by webcam (I called it: Blender Live Shape :])
I use Processing for capturing webcam and generating greyscale images. There is subdivided plane in Blender with displacement modifier. Python script periodically replace modifier texture with new image. Virtual serial port pair make synchronization. At last we receive object with real-time generated mesh. It works in Blender standard mode and in Game Engine. All source code is on my project website. I think it’s possible to use some python libraries for capture webcam direct to Blender.
Soon I’m going to make 3D scanner using webcam, Processing and Blender.
Links
Now that’s cool!!!!
Actually, this has been done in Quartz Composer/VVVV before. But its always good to see such features in a full fledged 3D application.
The more real-time software interconnection projects there are, the better. The prospect of true digital puppetry in Blender banks on it.
Ah, my two favorite programs, Blender and Processing!
Nice work!
—
Kevin
Wow !! Cool!
Wow, this is very impressive! Is it really possible to make a 3D scanner using this? If so, coupled with some retopo it would make a powerful tool.
ah!!!great!!!
Awesome work!
Blender and processing?most cool!!! Good luck with the aquiring the data for 3d scanning!
Genius work! Can I assume you may then apply the modifier at any time in order to “capture” the live shape? Good work. Keep it up! I’m very interested in seeing what you come up with for 3D “scanning”.
Nice! I see a high potential in this technique for other new future features … e.g., transforming existing objects in real time with your fingers … building your own worlds and objects not with a mouse or pen. Or something else :-)
Question- is there any other equipment involved, just trying to understand how you get the webcam to talk to blender, I assume you only need blender and a webcam to replicate this amazing experiment?
3D scanner using a webcam!!! that’s fantastic!
Genius idea! I would had never thought of doing that!
It’s the little side projects like this that make Blender so interesting to watch.
Cool, try to tune a bit the image using PIL in order to get less noise and more accuracy! Besides that, is an excellent idea!
how do you make it?
fantastic
Great experiment !
A simple to use 3D scanner usable with Blender would be a great tool !
By parenting a small cylinder, or any needle shaped object to the Displaced mesh and
by using duplivert, you can make animated nails boards !
Good work !
that is really cool stuff keep going!
Thanks friends! I’m very glad you like it. Replies for some comments:
Yes, we can add all modifiers we want and have fun with it.
Webcam, Processing and Blender, just it, see and try: http://en.myinventions.pl/018/BlenderLiveShapes.html
Yes, there’s some noise I can fix with smooth modifier, Processing filters (or better cam).
Yeees, 3d modelling with “webcam-pad” :)
It can be used as a 3d scanner, but object colors and nonperfect light sources make a lot of noise… so I working on scanner with laser pointer (but not real-time…)
I think it’s possible to make full python script for connect webcam direct to Blender…many many ideas, Blender is perfect tool!
Whaaa??!?!?! Amazing! How many poly’s on that plane?
Nooow that’s really cool…
I think blender won’t be open source anymore if there’s more powerful techniques like this oO
Wow…with a studio-type photography setup, this could save a lot of time.
There is 4225 vertex in mesh, and 200x150pixels webcam. Refresh rate: 20-26fps.
Cool, you could actually sculpt landscapes in Play-Doh instead of using a Wacom Tablet :)
It looks like some sort of freaky horror movie material. Of course it would be hard to use such a displacement system for anything useful because the objects are stuck in an otherwise flat surface. However, you could still do some interesting things, like a face looking out of a wall.
I wonder if it is able to record the mesh deformation as an animation?
Very excited about the 3D scanner project!!
Hot Damn!! Nice work!
@Experiment Garden:
Yeah! Just in time for Halloween!!
Putting a(ny) texture on the plane might make it more obvious what happens in the video. Interesting work :-)
I’m wondering if you could also simultaneously use the color image data from the camera to add the color layer on top of the displacement map, via nodes or other method. Kind of like layering skin, with diffuse, bump , specular and color maps. Then you’d have a real image, cool effect.
Cool and rough. Nice effect. Does anyone think of any applications?
I searched for posibility of recording deformable mesh…but I can’t find suitable python function… maybe in 2.5 will be.
There is problem with real-time refreshing texture.
Maybe standard webcam for capturing colorfull texture, and IR webcam with IR diodes for grayscale map…I will try
I searched for posibility of recording deformable mesh…but I can’t find suitable python function… maybe in 2.5 will be.
There is problem with real-time refreshing texture.
Maybe standard webcam for capturing colorfull texture, and IR webcam with IR diodes for grayscale map…I will try
OH! You’re my new favorite blogger fyi
I think it’s possible to make full python script for connect webcam direct to Blender…many many ideas, Blender is perfect tool!
Yes Sebastian, just use OpenCV+PIL and you’re done :) That’s the same thing I did when doing the object tracking in the BGE :)
All we need is a good automated retopo/ smoothing tool to get rid of noise afterwards.
Looks Fun, great idea, just getting into Processing myself.
A similar idea, video with Red, Green and Blue lights positioned around the subjects face to get a NOR map video? :-)
Hi Sebastian, this project is really useful, since it will allow to “track” relative vertex positions for facial animations, which is a real pain for me, but I see this tool of yours will help me a GREAT deal. Now, I am trying to install it in an acer netbook, and though I already installed the winvdig, it stops running and says that my camera had a problem trying to link the video output to the video renderer. Could you suggest a workaround for this?
Thanks a lot for enhancing our blender experience with your cutting-edge inventions!! Please, keep them coming ;)
que tal…
yo quiero uno de esos en casa :)
saludos…
Really really great. However, your instructions on the site aren’t very friendly. How do you setup com0com with the webcam? I just never used com0com=P
You should just instal Com0com and rename deafult CNCA0 and CNCB0 with COM10 and COM11. Processing send data to COM10 and Com0com send it from COM10 to COM11. Blender reads from COM11.
I wrote article about using wirtual serial ports, but just in Polish.
holycrap thats an awesome combination!!!
I wonder if one could randomize the displacement while it is created to achieve different effects with this,
like stirring your hands infront of the cam to achieve a vortex waterdrop like effect or just some random chaos:)
Great work, baffling result!
Try using 2 webcams to define points in space and reconstruct the object;-)
Something like this:
http://www.stanford.edu/class/cs229/proj2007/HuetIllowsky-SelfCalibrationofaPairofWebcamsforStereoVision.pdf
Anyway, great idea!
Awesome work!!! this method opens many possibilities, especially creating 3D scans without having to buy extremely expensive equipment.
My jaw litterally dropped straight to the ground when I was watching this video. I have GOT to get this!!!! :D
New Blender Live Shapes link: http://myinventions.pl/index.php?page=BlenderLiveShapes
Link updated, thanks.