3D shape reconstruction is a well-studied problem in Computer Vision literature and robust algorithms exist for 3D reconstruction using multiple photographs of rigid environments such as buildings and man-made objects. However, rigid 3D reconstruction using just one photograph is an ill-posed problem as we need to estimate 3D data from 2D observations. On the other hand, recovery of non-rigid 3D structure underlying human motion and clothing, from one or more videos is also a fundamentally challenging problem. In this thesis we propose novel geometric approaches for Single View Reconstruction using an abundance of orthogonal angles in urban environments, and non-rigid 3D reconstruction using limited spatiotemporal deformations due to natural motion constraints. Rigid man-made structures, such as buildings, are characterized by a profusion of mutually orthogonal line-pairs. Previous literature uses this regularity by grouping lines into orthogonal vanishing directions to rectify the projectively distorted images of planes. Unfortunately, this is a global constraint since all lines must follow a grid structure. Instead, we use locally adjacent orthogonal line-pairs for 2D Metric Rectification and demonstrate a robust solution wider applications. We pose the problem as explicit plane pose recovery which easily extends to Single View Reconstruction (SVR) of a multi-planar scene if plane boundaries are known. Moreover, we propose the first automated line based SVR algorithm with automatic segmentation for arbitrary plane and camera orientations. We only use the local angle regularity assumption combined with a 2.5D multi-planar layout. This is in contrast to the previous line-based algorithms that worked exclusively in either indoor or outdoor scenarios; restricted line, plane and camera orientations globally; and required ground plane or ceiling to be visible. We also extend the angle regularity idea into 3D where projectively distorted multi-planar structures, recovered using uncalibrated cameras, are rectified using locally adjacent orthogonal plane-pairs. Natural motion of humans, animals and clothing results in deformable shapes but these deformations are not arbitrary due to physical constraints. These regularities are typically posed as compactness of shape and trajectory bases so the deformable structure can be represented with far fewer parameters. This compact representation is useful when recovering non-rigid 3D structure using a single camera - otherwise an ill-posed problem since we need to recover 3D points from an equal number of 2D observations. Traditional algorithms require all feature points to be tracked at all times, stack them together in an observation matrix, and use matrix factorization to recover the 3D structure using rank constraints arising from shape or trajectory compactness. However, tracking all feature points at all times is not practical due to occlusions and deformations in the local patches being tracked. Therefore, we formulate these regularities as Local Rigidity constraints in space and time, resulting in a robust algorithm which reconstructs much shorter tracked sequences in the presence of missing data. In typical commercial applications known as Motion Capture, multiple static infrared cameras are required to track and reconstruct the deformable shapes. We extend the shape and trajectory compactness idea to multiple static cameras and propose an elegant factorization algorithm that works in the trajectory and shape subspaces directly. In contrast with previous single and multi-camera approaches, the proposed algorithm handles considerable amounts of noise and missing data, allowing for applications where specialized infrared cameras and markers are not available to aid in feature tracking. Ubiquitous nature of geometric regularities restricts the solution space for several geometric vision problems but investigating all possible applications of geometric regularities is intractable. Nonetheless, we present a wide range of novel algorithms for 2D, multi-planar 3D, and non-rigid 3D problems, using one or more cameras, while extending beyond the stability and applicability of previous solutions. We believe this provides sufficient evidence in favor of using geometric regularities for shape recovery.