Learning Non-Rigid 3D Shape from 2D Motion

Lorenzo Torresani, Aaron Hertzmann,Chris Bregler

We present an algorithm for learning the time-varying shape of a non-rigid 3D object from uncalibrated 2D tracking data. We model shape motion as a rigid component (rotation and translation) combined with a non-rigid deformation. Reconstruction is ill-posed if arbitrary deformations are allowed. We constrain the problem by assuming that the object shape at each time instant is drawn from a Gaussian distribution. Based on this assumption, the algorithm simultaneously estimates 3D shape and motion for each time frame, learns the parameters of the Gaussian, and robustly fills-in missing data points. We then extend the algorithm to model temporal smoothness in object shape, thus allowing it to handle severe cases of missing data.


paper: Learning Non-Rigid 3D Shape from 2D Motion, Lorenzo Torresani, Aaron Hertzmann and Christoph Bregler, NIPS 2003 (pdf)

matlab software: download the software here.
Follow the included readme.txt  for help. You are strongly recommended to read the paper if you are interested in understanding the code and experimenting with it.
Let us know if you get cool results or if you end up using this software for your research!

video: quicktime video of the the shark reconstruction (same as the animated gif above)

back to the movement group page