Blender Live Shape – Real-Time Displacement Modifier

Posted on by

Sebastian Korczak likes to hook up gear to Blender. Check out his latest experiment: Blender Live Shape – a real-time displacement modifier (similar to normal map) based on live grayscale video from webcam. Read on for more of his work.

Sebastian writes:

For years I’m interested in Blender but this year I started to make some electronic project connected with Blender. Probably you like to see my concept of real-time 3D object in Blender controlled by webcam (I called it: Blender Live Shape :])

I use Processing for capturing webcam and generating greyscale images. There is subdivided plane in Blender with displacement modifier. Python script periodically replace modifier texture with new image. Virtual serial port pair make synchronization. At last we receive object with real-time generated mesh. It works in Blender standard mode and in Game Engine. All source code is on my project website. I think it’s possible to use some python libraries for capture webcam direct to Blender.

Soon I’m going to make 3D scanner using webcam, Processing and Blender.

Links

Be Sociable, Share!

    Twitter



    Please support us

    • http://www.dustyghost.com dusty

      Now that’s cool!!!!

    • http://www.gutsblow.com/ Satya Meka

      Actually, this has been done in Quartz Composer/VVVV before. But its always good to see such features in a full fledged 3D application.

    • http://www.rgbk.org wraybowling

      The more real-time software interconnection projects there are, the better. The prospect of true digital puppetry in Blender banks on it.

    • http://www.virtualmatter.org KevinW

      Ah, my two favorite programs, Blender and Processing!

      Nice work!

      –
      Kevin

    • 3sie 3wiel

      Wow !! Cool!

    • Nik

      Wow, this is very impressive! Is it really possible to make a 3D scanner using this? If so, coupled with some retopo it would make a powerful tool.

    • Saverio

      ah!!!great!!!

    • Jeff

      Awesome work!

    • txrx

      Blender and processing?most cool!!! Good luck with the aquiring the data for 3d scanning!

    • http://www.dynavisionfx.com Bart Robinson – VFX Artist

      Genius work! Can I assume you may then apply the modifier at any time in order to “capture” the live shape? Good work. Keep it up! I’m very interested in seeing what you come up with for 3D “scanning”.

    • Marotzke

      Nice! I see a high potential in this technique for other new future features … e.g., transforming existing objects in real time with your fingers … building your own worlds and objects not with a mouse or pen. Or something else :-)

    • Ian3d

      Question- is there any other equipment involved, just trying to understand how you get the webcam to talk to blender, I assume you only need blender and a webcam to replicate this amazing experiment?

    • bob2

      3D scanner using a webcam!!! that’s fantastic!

    • redbyte

      Genius idea! I would had never thought of doing that!

      It’s the little side projects like this that make Blender so interesting to watch.

    • http://www.aktasway.com Akta

      Cool, try to tune a bit the image using PIL in order to get less noise and more accuracy! Besides that, is an excellent idea!

    • http://nikitronn.narod.ru nikitron

      how do you make it?
      fantastic

    • http://3d-synthesis.com ROUBAL

      Great experiment !

      A simple to use 3D scanner usable with Blender would be a great tool !

      By parenting a small cylinder, or any needle shaped object to the Displaced mesh and
      by using duplivert, you can make animated nails boards !

      Good work !

    • kABHIr

      that is really cool stuff keep going!

    • http://en.myinventions.pl Sebastian

      Thanks friends! I’m very glad you like it. Replies for some comments:

      Yes, we can add all modifiers we want and have fun with it.

      Webcam, Processing and Blender, just it, see and try: http://en.myinventions.pl/018/BlenderLiveShapes.html

      Yes, there’s some noise I can fix with smooth modifier, Processing filters (or better cam).

      Yeees, 3d modelling with “webcam-pad” :)

      It can be used as a 3d scanner, but object colors and nonperfect light sources make a lot of noise… so I working on scanner with laser pointer (but not real-time…)

      I think it’s possible to make full python script for connect webcam direct to Blender…many many ideas, Blender is perfect tool!

    • http://www.robo3d.com Robo3Dguy

      Whaaa??!?!?! Amazing! How many poly’s on that plane?

    • Jack*RED

      Nooow that’s really cool…

      I think blender won’t be open source anymore if there’s more powerful techniques like this oO

    • Alex Blank

      Wow…with a studio-type photography setup, this could save a lot of time.

    • http://en.myinventions.pl Sebastian

      There is 4225 vertex in mesh, and 200x150pixels webcam. Refresh rate: 20-26fps.

    • Marc

      Cool, you could actually sculpt landscapes in Play-Doh instead of using a Wacom Tablet :)

    • http://experimentgarden.blogspot.com Experiment Garden

      It looks like some sort of freaky horror movie material. Of course it would be hard to use such a displacement system for anything useful because the objects are stuck in an otherwise flat surface. However, you could still do some interesting things, like a face looking out of a wall.

      I wonder if it is able to record the mesh deformation as an animation?

    • Neil

      Very excited about the 3D scanner project!!

    • http://weber.edu/planetarium Ron Proctor

      Hot Damn!! Nice work!

      @Experiment Garden:

      Yeah! Just in time for Halloween!!

    • yoff

      Putting a(ny) texture on the plane might make it more obvious what happens in the video. Interesting work :-)

    • http://www.animationbytim.com/ Tim Formica [Room335]

      I’m wondering if you could also simultaneously use the color image data from the camera to add the color layer on top of the displacement map, via nodes or other method. Kind of like layering skin, with diffuse, bump , specular and color maps. Then you’d have a real image, cool effect.

    • ralmon

      Cool and rough. Nice effect. Does anyone think of any applications?

    • http://en.myinventions.pl Sebastian

      I searched for posibility of recording deformable mesh…but I can’t find suitable python function… maybe in 2.5 will be.

      There is problem with real-time refreshing texture.

      Maybe standard webcam for capturing colorfull texture, and IR webcam with IR diodes for grayscale map…I will try

    • http://www.stacyinshape.com/ Sebastian

      I searched for posibility of recording deformable mesh…but I can’t find suitable python function… maybe in 2.5 will be.

      There is problem with real-time refreshing texture.

      Maybe standard webcam for capturing colorfull texture, and IR webcam with IR diodes for grayscale map…I will try
      OH! You’re my new favorite blogger fyi

    • http://www.aktasway.com/blog Akta

      I think it’s possible to make full python script for connect webcam direct to Blender…many many ideas, Blender is perfect tool!

      Yes Sebastian, just use OpenCV+PIL and you’re done :) That’s the same thing I did when doing the object tracking in the BGE :)

    • Marcus

      All we need is a good automated retopo/ smoothing tool to get rid of noise afterwards.

    • yellow

      Looks Fun, great idea, just getting into Processing myself.

      A similar idea, video with Red, Green and Blue lights positioned around the subjects face to get a NOR map video? :-)

    • http://www.facebook.com/home.php#/roblesluis?ref=name Luis Robles

      Hi Sebastian, this project is really useful, since it will allow to “track” relative vertex positions for facial animations, which is a real pain for me, but I see this tool of yours will help me a GREAT deal. Now, I am trying to install it in an acer netbook, and though I already installed the winvdig, it stops running and says that my camera had a problem trying to link the video output to the video renderer. Could you suggest a workaround for this?

      Thanks a lot for enhancing our blender experience with your cutting-edge inventions!! Please, keep them coming ;)

    • mercury

      que tal…
      yo quiero uno de esos en casa :)
      saludos…

    • starplayer

      Really really great. However, your instructions on the site aren’t very friendly. How do you setup com0com with the webcam? I just never used com0com=P

    • http://en.myinventions.pl Sebastian

      You should just instal Com0com and rename deafult CNCA0 and CNCB0 with COM10 and COM11. Processing send data to COM10 and Com0com send it from COM10 to COM11. Blender reads from COM11.

      I wrote article about using wirtual serial ports, but just in Polish.

    • Nixon

      holycrap thats an awesome combination!!!
      I wonder if one could randomize the displacement while it is created to achieve different effects with this,
      like stirring your hands infront of the cam to achieve a vortex waterdrop like effect or just some random chaos:)

      Great work, baffling result!

    • Wim

      Try using 2 webcams to define points in space and reconstruct the object;-)

      Something like this:
      http://www.stanford.edu/class/cs229/proj2007/HuetIllowsky-SelfCalibrationofaPairofWebcamsforStereoVision.pdf

      Anyway, great idea!

    • Rave_Zero

      Awesome work!!! this method opens many possibilities, especially creating 3D scans without having to buy extremely expensive equipment.

    • http://www.twitter.com/goldenacorn93 goldenacorn93

      My jaw litterally dropped straight to the ground when I was watching this video. I have GOT to get this!!!! :D

    • http://en.myinventions.pl Sebastian
    • http://www.blendernation.com Bart

      Link updated, thanks.