While both work with images, Computer Graphics and Computer Vision are different fields. Computer graphics starts with the creation of geometric models and produces image sequences. Computer vision starts with images or image sequences and produces interpretations including geometric models. Lately, there has been a meeting in the middle, the goal being to create photorealistic images with the help of accurate models recovered from projections of real objects. This convergence has produced a new subfield called Image-Based Rendering and Modeling (IBRM). In this research, the geometric aspects of IBRM are studied. The case of two views is studied first. New algorithms are developed that automatically align the input images, match them and reconstruct 3-D surfaces in Euclidean space. The matching algorithm is designed to cope with complex shapes such as human faces. The reconstruction algorithms are then generalized to the multi-view case, based on a stratified framework. At the root of the stratification is a novel projective reconstruction algorithm that produces a non-metric structure of the scene. An image-based rendering system is implemented, that directly uses this non-metric structure to synthesize images from novel viewpoints. On top of that, a novel algorithm is developed to upgrade the non-metric structures into metric ones. Intrinsic and extrinsic camera parameters are obtained at the same time. One potential application, showcased in this thesis, is to blend computer graphics objects into real images with correct perspective and occlusion. The proposed theory and the associated algorithms lay down the groundwork for future development in multi-view image based rendering and modeling.