To cope with the exponential growth in telecommunication traffic, next generation optical networks emerge as viable solution due to its ability of unlimited bandwidth , high data rates and low cost. Wavelength division multiplexing (WDM) is the core technology used in the long haul high capacity optical transmission. In long-haul high capacity optical transmission systems, the signal degrades due to two major impairments of linear and non-linear issues which severely degrade the transmission performance of the optical transmission system. The non-linearity of optical transmission is arguably the most significant factor which is limiting the capacity of modern-wavelength division multiplexing (WDM) optical network- system. The nonlinear effect of the fiber introduces complex distortions into the transmitted optical- waveforms, producing non-linear interference among WDM channels, which diminishes the receiver’s capability, limits the transmission reach and achievable data rates. The infrastructure of non-linear factors and understanding of their impacts on optical system performance are therefore key components in the efficient design of long-haul, high capacity and high data rate optical fiber communications system. In this dissertation we formulate linear and non-linear impairments for long-haul and high capacity optical transmission network and also to develop a less-complex and cost-effective model of optical-transmission based on mathematical linear and non-linear interference. The postulatory model is validated by using simulation of key non-linear factors such as effective area, launch power, nonlinear refractive index and fiber length. The transmission performance of the high-capacity long-haul optical network has been analyzed on the basis of some key parameters such as bit error rate and optical signal to noise ratio. Mitigation of non-linear impairments shows significant impact on transmission performances of high capacity long-haul optical networks. iv The impact of major non-linearity in optical fiber such as SRS (Stimulated Raman Scattering) and Four Wave Maxing (FWM) in wavelength division multiplexing optical long-haul and high speed data rate transmission scheme have been investigated at different data rates. The proposed model is analyzed based on different data rate such as 10Gbps, 40Gbps and 100Gbps per channel. The research work has also investigated transmission performances for different modulation formats such as Non Return to Zero, Return to Zero, Carrier Suppressed Return to Zero and Duo binary RZ formats. 16*100 Gbps multichannel technique is used with 200GHz channel spacing. The simulations are analyzed for different values of input power, length of fiber, non linear refractive index, non-linear dispersion and non-linear effective area for all above mentioned modulation formats. To evaluate the effect of modulation format Fiber Bragg Gratting, optical fiber amplifier and Dispersion Compensation Fiber techniques are enacted on the proposed optical network system. Moreover the thesis has investigated one of the major non-linear effects of FWM using the oretical model and validated through simulation in Optisystem. Two mechanisms with and without dispersion variation are investigated. FWM product for various values of launch power is investigated. The transmission performances of the proposed model is evaluated on the ba sis of Bit Error Rate (BER) , Optical Signal to Noise Ratio (OSNR) and quality factor using key parameters of effective area, non-linear refractive index, non-linear dispersion and linear dispersion. Fiber-optic communication systems predominantly when employ in the non-linear regime, generally do not perform truly as theory would speculate. Hence the transmission performance of the system also explored based on power budget, power penalty and cost benefit analysis. The presume model is affirmed using important non-linear factors like effective area, launch power, received power, refractive index and length of the fiber. The power budget and transmission performance of the high capacity long haul optical network are analyzed on the basis of bit error v rate and optical signal noise to ratio against launch power, length of the fiber, effective area and non-linear refractive index. The simulation results manifest significant impact of non-linear impairments on power budget, cost benefit analysis and transmission power penalties of the high capacity long-haul optical networks.