The investigation of electrostatic excitations and associated instabilities at the ionic timescale in ultra-high density (degenerate) quantum plasmas plays a fundamental role in understanding the collective interactions in superdense astrophysical systems, such as in white and brown dwarfs, in magnetars, in neutron stars as well as in laboratory such as in ultraintense laser-matter interaction experiments. In this context, a generalized relativistic Chandrasekhar equation of state (EoS) is employed for inertialess degenerate species (electrons/positrons), while the ions are taken as non-degenerate and inertial. The relativistic degeneracy effects in dense astrophysical plasmas play a vital role on the collective dynamics of degenerate dense plasmas. In this thesis, we primarily focus on the analytical and numerical study of linear and nonlinear ion-acoustic (IA) excitations in degenerate electron-positron-ion (e-p-i) dense plasmas, and in particular four different features are investigated. The first investigates the Korteweg-de Vries (KdV) and Korteweg-de Vries Burgers (KdVB) equations through the well-known reductive perturbation technique. Smaller (in amplitude) and narrower (in width) IA solitons are obtained for increasing values of relativistic degeneracy parameter and positron concentration, while taller and steeper shocks result for higher values of relativistic degeneracy parameter, positron content and ion kinematic viscosity. The second further extends the study to develop and investigate Zakharov-Kuznetsov (ZK) and Zakharov-Kuznetsov Burgers (ZKB) equations governing the three dimensional propagation of IA solitons and shocks in a magnetized degenerate e-p-i dense plasma. It is shown that the characteristics of IA solitons and shocks are substantially influenced by the intrinsic plasma parameters (i.e., the relativistic degeneracy parameter, the positron content, the ion gyrofrequency and the direction cosines). The third application is a comparative study of small amplitude and arbitrary amplitude IA waves in a degenerate e-p-i dense plasma with finite ion temperature effects. Concentrating on large amplitude IA excitations, the fluid equations are scaled and reduced to obtain an energy-balance equation in terms of the Sagdeev potential function. It is shown that the small amplitude expansion of the Sagdeev energy balance equation gives exactly the same result as predicted by the KdV theory, for pulses moving at a weakly supersonic velocity. The range of allowed values of the soliton speed (viz., the minimum and maximum Mach numbers), wherein solitary waves may exist, is determined. The impact of the key plasma configuration parameters, namely the electron relativistic degeneracy parameter, the ion temperature-to-electron Fermi temperature ratio and the positron content, on the soliton characteristics and existence domain, is examined numerically. The fourth involves a detailed theoretical and numerical analysis of nonlinear amplitude modulation of IA waves in a degenerate warm e-p-i plasma. By employing the multi- scale perturbative method, the nonlinear Schrödinger equation (NLSE) for the envelope amplitude is derived, based on which the modulational instability (MI) is also examined. In particular, the relativistic degeneracy parameter has the effect of attenuating the MI. It is shown that various types of localized IA excitations exist in the form of bright type solitons (envelope pulses) or dark type solitons (voids). The relativistic degeneracy parameter, the positron concentration and ion thermal effects significantly modify the occurrence conditions for MI, the associated threshold, the growth rate as well as the characteristics features (amplitude, width) of envelope solitary structures.