Below is a list of research projects that I completed in the past few years.

Projects completed at University of Southern California:

My Ph.D. research addresses a fundamental issue in computer vision: what can we get from a video sequence which is shot by a moving camera observing a scene with moving objects? My goal is to infer the complete 2D and 3D shape of both moving objects and static background from these kinds of video sequences.

The 2D shape includes the motion blobs, which correspond to moving objects, and the rest image areas, which are the static background. This is indeed a motion segmentation process, classifying each pixel into either motion blobs or background area. The segmented motion blobs are further tracked across multiple frames to form object trajectories.

Ideally, the 3D shape inference refers to computing the depth for each pixel in the image, which may belong to either motion blobs or static background. However, this is not always feasible for the real-world video sequences. A practical solution to 3D shape inference is to start from the sparse set of points and lines on the object surface, a typical "Structure from Motion" process. Then the sparse structure is extended to denser representation of object shape. One approach is to decompose the 3D object volume into a set of elements, termed as voxels. A volumetric reconstruction process is performed to determine which voxels are occupied by the objects. Then the 3D object shape is obtained by converting voxel-based representation into surface-based representation.

The complete process of 2D and 3D shape inference process is roughly divided into three stages as listed below:

Projects completed at Microsoft Research Redmond and Microsoft Research Asia:

Projects completed at Tsinghua University: