This project is devoted to research into combination of sketch-based modeling and computer vision techniques for reconstructing complex real-world objects from videos. This project was started in 2004 in cooperation with Samsung Advanced Institute of Technology.

Despite serious progress in the field of shape-from-X reconstruction techniques, fully automated reconstruction of arbitrary real-world objects still fail to yield acceptable result over a broad class of object shapes, image types and scene complexity. Hence, research into efficient and easy-to-use combination of automated and manual 3D modeling from images is very promising.

The project key idea is to support computer vision techniques with user interaction to contribute to overall scheme robustness. The project goals include:

1. User-assisted creation of 3D models of objects from the raw 3D data (dense depth maps, sparse 3D points);
2. Development of the UI mechanisms for interactive 3D object shape reconstruction support. This includes the general paradigm of user intervention into reconstruction process and efficient ways for its implementation;
3. Enhancement and adaptation of the 3D reconstruction methods to the needs of interactive reconstruction;

Images from the input sequence of 25 frames

Reconstruction

The overall reconstruction process comprises of the following steps:

1. Images acquisition:
   • Acquiring object images with a hand-held camera;
2. Structure and motion estimation:
   • Feature tracking;
   • Structure and motion (SAM) initialization;
   • Sequential SAM estimation by:
     1. Pose estimation for each camera;
     2. Structure update based on current camera pose;
     3. New structure initialization;
   • Pose refinement of all cameras;
3. User-guided basic geometric shapes fitting to sparse 3D points:
   • Segmenting object 3D points from background;
   • Guided fitting of geometric shape (cylinder, plane, polyhedron, surface of revolution);
4. Sketch-based editing of reconstructed object
   • Adding missed detail;
   • Correcting reconstruction errors;

Tracked features, estimated 3D points and camera movement
Cup handle reconstruction from 2 sketched silhouettes

Results

Examples of several reconstructed objects are shown below. Pay attention to complex cup handle shape captured precisely.

The project team

Principal researcher - Vladimir Vezhnevets (vvp @ graphicon . ru)

Researchers: Anton Konushin, Evgeny Lisitsin, Viktor Gaganov, Evgeny Seredenko, Anton Yakubenko.

Collaborators at the SAIT side: Keun Ho Kim (Multimedia Lab, SAIT)

Contacts