Reconstrucing the 3-D Scene from a Single Intensity 2-D Image

CSCI 574 Computer Vision Project

Elaine (Eun Young) Kang : elkang@iris.usc.edu


[Introduction] [Approach] [Result ] [Analysis]


Introduction
This project is to recover 3-D information from a 2-D image, and deals with real images. Given the image of a known object, the calibration cube, taken by an unknown camera from an unknown viewpoint, we need to calibrate and to relate the world coordinate system with the camera coordinate system. That is, based on camera calibration from calibration cube, we generate 3D wireframe model the image of a scene, taken from the same viewpoint by the same camera.


Approach ( workflow)

  • Camera calibraion

    Tsai's method
    Among 4 options of Tsai' methods, I built non-coplanar non-opimization method using 7 points. -> Result : not so good. (around 25 pixels errors)

    Solving the system of linear equations based on hw#2
    Use 7 points -> Result : better than Tsai's method (around 5~13 pixels errors) So for the project, I used the matrix from this method to get world coordinates from image coordinates.

  • Preprocesing : massage input edges from VCanny

    - Remove insignificant edges which is either very short or isolated from everything else.
    - Do the linearlization using inner_product. If the angle between two adjacent edges is almost 180, consider the two lines are colinear.
    - Merge corner : First, build quantization map. If the corners are very close based on the map, merge together.

  • Detect Top face

    1. Gather all candidate edges of top faces
    - Remove vertical edges and gather all other edges (This is first reason that my program doesn't work at all for the first view of second level, since the top edges of the image are almost vertical).

    2. Find top corners - Select two adjacent edges among top edges. If the lines are not parallel each other, get the crossing point. If the crossing point is near end_points of the two edges, link them together and adjust the end_points.

    3. Top faces - Among the top edges, try to find edge paths.

    If the result is 4 edge cycle, assume it as a top_face and check if it makes paralleogram.
    If the result is 4 edge non-cycle, try to link the open edge and check if it makes paralleogram.
    If the result is 3 edge non-cycle, link the ends of the edges and check if it makes parallelogram.
    For all cases, they make almost parallelogram, hypothesize it top-face.
    If the result is 2 edge path, check if two edges are perpendicular.
    If they are, make one additonal parallelogram and hypothesize it top-face.

  • Detect Vertical face

    1. Vertical edges
    - gather all vertical edges and sort them in the incresing order of X coordinate.

    2. Verify Top face
    - Count the number of vertical edges which are adjacent to one of top face points and all Y coordinate values are lower than the point.
    If the number of vertical edges are greater than 1, try to merge the end_points.
    Mark the top face is verified and mark block top face is known.
    (This is other reason that my program didn't work at all for the 2nd view of the 2nd level image. Since the input edges are not fully linked with top edges or even dosn't look vertical, it didn't partially verify top faces.)

    3. Vertical Faces
    - With gathered vertical_edges, generate all possible vertical faces.

    4. Verify vertial face
    - Check if there is a top face which shares at least 1 points with upper points of vertical face.
    - If there is, check if the vertical face is paralleogram.
    If it doesn't make it, trim the length based on the longest one (use top_face slope), and make it parallelogram.
    Mark the vertical face is verified.
    - copy vertical faces and corrssponding top face into block and mark vertical.

  • Recover bottom face

    1. Bottom edges
    - gather all other edges which are not used so far.

    2. Bottom faces
    - In case that the vertical face is known, make vertical length equal based on the lonest one.
    - If there exists the bottom edges which is neighbor to the lower vertical edges of each vertical face, adjust the bottom edges.
    - Recover bottom face using top face information.

  • Height Recovery & Generate 3D coordinates

    - From Z value of one point of bottom face Z(Z=50) and the corresponding point of top face, we can get the height difference. Using the ratio of pixel to mm, get the Z value of the top face.
    - From the (u,v), image coordinate, and Z, we get (x,y) using the calibration matrix.


    Result
    My program only reconstructs two images in the first level. I failed to enhance, and I think that I made worse and wasted time. (Hope to enhance near future. )

    1. 3D block from the first view image

    2. 3D block from the second view image


  • Analysis
    Difficulties
    - There were many difficulties. Especially, I had big trouble with setting threshold values. I found algorithm is very very sensitive to threshold values.
    - Time constraints.

    Lesson
    - Vision is difficult, but it is interesting domain.
    - Don't try to change your program right before due date.