The propagation characteristics of nonlinear coherent structures involving the solitons, shocks and rogons (rogue waves) are studied in a degenerate dense plasma, which has strong relevance not only to compact astrophysical bodies such as white dwarfs, magnetars, neu tron stars, etc., but also in high density laser-induced plasmas. In superdense plasmas, the quantum corrections due to quantum degeneracy, quantum diffraction, relativistic degener acy and degenerate particle trapping strongly modify the nonlinear properties of the waves and instabilities. In Chapter 3, a multi-dimensional propagation and stability for ion-acoustic (IA) soli tary waves are discussed in a magnetoplasma, containing degenerate relativistic trapped electrons and classical anisotropic ions. By using the Chew Goldberger and Low (CGL) the ory alongwith quantum magnetohydrodynamic (QMHD) equations, a Zakharov-Kuznetsov (ZK) equation is obtained, which admits a solitary solution in the form of potential pulse. The stability parameter Si is analyzed by a k-expansion method, which reveals the stable and unstable solitons in the domains Si < 0 and Si > 0, respectively. Numerically, it is found that trapped electrons and ionic pressure anisotropy support the formation of taller and wider solitons in a degenerate relativistic anisotropic plasma. Chapter 4 presents the linear and nonlinear analyses of magnetosonic (MS) solitary and shock waves in a degen erate relativistic magnetoplasma. For this, we have solved the QMHD equations with the aid of reductive perturbation technique to study a Korteweg-de Vries (KdV) solitons and Korteweg-de Vries-Burger (KdVB) shocks. For linear study, a plane wave solution is used to obtain a linear dispersion relation for MS waves. It is revealed that relativistic electrons reduce the rapid variations in the wave frequency due to intensification of external mag netic field. In nonlinear analysis, the relativistic electrons strongly favor to excite taller and narrower localized solitons and amplify the shock strength. In Chapter 5, one-fluid QMHD model is utilized to investigate the characteristics of low frequency MS envelope solitons in a dense magnetized plasma. By using a multiscale reductive perturbation technique, the nonlinear Schro¨dinger (NLS) equation is derived to account for degenerate relativistic electrons and associated quantum diffraction effects. It is examined that these effects play a vital role to excite modulationally stable excitations, such as the dark and grey solitons in dense magnetoplasma. These solitons only exist under the condition PQ > 0, where P and Q are the dispersion and nonlinearity coefficients, respectively. These results are useful for analyzing the propagation characteristics of MS excitations in dense astrophysical plasmas, where quantum and relativistic effects modify the wave properties and stability. Chapter 6 describes the evolution of weakly nonlinear dust-ion-acoustic (DIA) bright solitons and rogons in a degenerate dusty plasma containing degenerate relativistic electrons, and dynamical degenerate ions with static dust. To examine numerically the modulational instability (MI) and unstable modes on ion-timescales, the effects of relativistic electrons and degenerate ions are examined on the profiles of MI and localized DIA excitations (bright solitons and rogons). It is found that the instability domain becomes widen for varying the relativistic parameter and ionic degeneracy. Finally, in Chapter 7, we have extended our work to nonlinear evolution of extremely low frequency dust-acoustic (DA) solitons and rogons in a nonplanar cylindrical plasma accounting for Thomas-Fermi distributed electrons and ions. The fluid equations are evolved into the cylindrical Kadomtsev-Petviashvili (CKP) and cylindrical nonlinear Schro¨dinger (CNLS) equations for studying the formation of solitary and rogue excitations. It is found that cylindrical plasma leads to radial deviation of solitons and modification of the growth rate associated with modulationally unstable wavepackets. The results are important to understand the nonlinear DIA and DA potential excitations in dense dusty plasmas where dust particulates modify the characteristics scales and affect the properties of stable and unstable wave modes.