Spectrum sensing and dynamic resource allocation has gained remarkable consideration from research community during the past decade in the field of cognitive radio networks. Moreover, femtocell networks are also an emerging technique which are deployed at homes in order to deal with growing demand of high rate data streaming and good voice quality. Cognitive femtocell incorporates the features of cognitive radios including spectrum sensing and resource allocation in order to mitigate interference and provide better utilization of the valuable licensed spectrum. In this dissertation we have presented certain spectrum sensing and resource allocation techniques in the area of both cognitive radios and cognitive femtocells. This dissertation presents a centralized relay-based spectrum sensing technique using hybrid nature based algorithm that detects the number of active primary users and jointly estimates (amplitude, carrier frequency and direction of Arrival (DOA) of far-field sources impinging on Uniform Linear Arrays (ULA) mounted on each relay. The fitness function is a sum of Mean square error (MSE) and the correlation error. However, the parameters are estimated by using Genetic Algorithm (GA) hybridized with Pattern Search (PS). Furthermore, an OFDM system with multiple users having heterogeneous traffic requirements is inspected. The goal is to simultaneously, maximize the capacity of all the Secondary User Equipment (SUE) with non-real traffic and to fulfill the minimum data rate requirement of real time SUEs, while keeping the interference to the Primary User Equipment (PUE) less than the predefined threshold value. In order to solve this non-convex problem, we have employed Artificial Bee Colony algorithm. Cognitive femtocell has emerged as a promising technique for indoor users in wireless communication systems since they have the cognitive and self-configuration capabilities. A number of methods have been used for spectrum sensing in femtocells with a cognitive engine. Another novel technique proposed in this thesis is near field source localization technique through which the cognitive femtocell will be able to detect the active femtocells in the licensed band and will get additional information including the range, amplitude, angle and operating frequency of a certain femtocell by employing genetic algorithm. Another contribution of this thesis is joint beamforming and null steering to maximize the signal to interference plus noise ratio (SINR) towards the secondary user equipment (SUE) while minimizing the interference towards the primary Base Station (PBS) in a cognitive femtocell network. We are reutilizing the sensed uplink (UL) frequency of the Primary users for the downlink (DL) transmission; the cognitive femtocell base station (CFBS) calculates such weight vector that aims at sending the maxima towards the femto user and null in the direction of PBS. In order to validate the implication and effectiveness of the proposed techniques, simulations have been carried out under different constraints and a definite assortment of different parameters.