Sign language is the language of gestures. It is also a way of communication for the deaf community. Sign languages use visual pattern rather than verbal communication. Sign Language Recognition (SLR) is an active research area in computer science. It has its roots in the domain of gesture recognition, robotics, gesture-based user authentication, lie detection communication, entertainment, security, art, industry, and sports. Every region has its own sign language. Pakistani Sign Language (PSL) for Urdu language is a visual-gestural language that is being used for communication by the deaf community. This research presents a robust, reliable, systematic and consistent system for both static and dynamic gestures. The present research focuses on different available solutions for gesture recognition and concludes that deep learning and convolutional neural networks give a most appropriate solution. The thesis is based on the comparison of different potential sign descriptors. Use of correlation and cross-correlation to identify gestures has led the researcher to the fact that supervised learning techniques have given a convincing performance for PSL or even any other sign language. The research has proposed 3 major ideas, it starts with proposing universal sign language and use of spelling-based gestures over word-based gestures. The research also proposes a video summarization technique for sign languages based on mean and entropy. Moreover, there is no standard dataset available for PSL, so dataset for a subset of static and dynamic signs of PSL is developed for the thesis. The research gives upto 90% accuracy when the recognition routine uses deep learning based model. The dataset is kept as small as 400 images/videos per class. The research has proven that the accuracy can be improved by increasing dataset size, image size, and number of epochs.