Fifth Generation (5G) is considered to be the future advancements in wireless technology. The ever increasing number of smart network devices may obsolete latest 4G technology for handling bandwidth allocation to a large number of smart devices. To meet with this challenging need, for rapid and efficient data transfer over such devices, requires next generation mobile technology. 5G technology is anticipated to offer appropriate solutions to such issues. High data rates can be achieved by using 5G, however, to make such futuristic technology realistic, is a real challenge. This challenging goal can be achieved by efficient utilization of the bandwidth in allocated spectrum. In this context, Cognitive Radio (CR), is an intelligent radio which has inherent adaptive procedures. It reuses the frequency when primary user is absent and works on principle of dynamic frequency allocation. CR is one of the key enabling technology for 5G networks which allows nodes to evaluate and manage underused licensed channels. It has brought benefits to a CR based 5G cellular networks which include ability to adapt dynamicity of a network, deals with the spectrum scarcity issue and operates in heterogeneous environment. It also provides intelligent solutions and self-governing capabilities to assist 5G key functions especially in smart beamforming. Another promising enabling technology of 5G is massive multiple-input multipleoutput (MIMO). Due to its attributes such as increased spectral and energy efficiency, it is envisaged as a potential technology for fifth generation (5G) wireless communication networks. As already mentioned that radio spectrum has become a scarce resource in wireless communications, hence imposes excessive cost on the high data rate transmission. Several linear and non linear detection techniques such as Zero-Forcing (ZF), Minimum Mean Square Error (MMSE), and Vertical Bell-Labs Layered Space Time (VBLAST) have been introduced. The purpose of such schemes is to mitigate the signal detection problems which are based on trade-offs between the bit error rate (BER) performance and computational complexities. The challenge in the design of massive- MIMO systems is developing less complex and efficient detection algorithms. The problem in building a receiver for massive-MIMO is to de-correlate the spatial signatures on the receiver antenna array. In this thesis, firstly, 5G technology, its challenges and proposed solutions are presented. Secondly, CR based 5G technology is investigated with the view to explore both technologies jointly and to present a comprehensive overview which will invigorate new research initiatives in this exciting field. Thirdly, a novel algorithm viz: Hybrid n-Bit Heuristic Assisted-VBLAST (HHAV) to perform an optimum decoding for massive-MIMO, is proposed. This structure is simulated in dynamic Rayleigh fading channel. One of the latest decoding algorithm viz: approximate message passing (AMP) is also evaluated with two threshold functions which include AMP with ternary distribution (AMPT) and AMP with Gaussian distribution (AMPG). Numerical results confirm that proposed HHAV algorithm performs significantly well as compared to the performances of aforementioned detection systems introduced in recent years. Finally, a novel 6G intelligent IOT paradigm is presented to optimize communication channels and process big data intelligently. The enabling technologies and AI based 6G air interface architecture, based on our proposed paradigm are also discussed. The results are evaluated through simulation by comparing our proposed paradigm with 5G-IOT and 5G intelligent IOT. In relation to our proposed concept, simulation results confirm that the performance in terms of effective utilization of channels is significantly improved.