The growth in web traffic has driven the rise in demand for information measure and high data rates. ‘Advance optical modulation formats’ are taken into account as a promising feature to satisfy the enhanced demand for information measure in broadband and other type of services. Lots of modulation formats have been proposed to increase the data rates but security implementation in optical communication system explicit to these modulation formats is still an open area to explore. In this thesis, the performance of some advance modulation formats as applicable to chaotic/secure optical communication is analyzed. The research work includes the generation of high bandwidth optical chaos through semiconductor lasers by controlling different parameters to get the optimum chaos. Initially, the basic chaotic communication system by synchronizing transmitted and received chaos is developed and tested for ordinary modulation schemes like RZ and NRZ to check validity of the scheme. Later, the long haul secure communication models based on Duobinary modulation and QAM format are designed and their performance is tested against various parameters. The Duobinary format is chosen due to its effective bandwidth utilization and robust performance against the intersymbol interference whereas QAM scheme is selected due to its better support against high data rates. In comparison to chaos produced by fiber lasers which are inherited by low bandwidth, semiconductor laser generated chaos is exploited for embedding the message. The performance of modulation schemes is studied for long distance communication models including non-linear and higher order effects. Various simulations experiments are conducted on latest versions of Optisystem 14.0 and MATLAB. This work can be vii considered as a first effort related to security implementation explicit to the advance modulation formats using optical chaos.