The study explores the evaluation criteria of individual and groups of raters as well as computes their inter rater and intra rater reliability on a national level high stakes examination in the Punjab province of Pakistan. The study thus looks at the process (the evaluation criteria) and the product (actual scores).Out of a total of nine examination Boards in the Punjab, three are in the South Punjab (SP). 34 raters from Multan, and 30 from DG Khan and Bahawalpur Board each totaling 94 markers, evaluated 20 essays twice after a gap of at least six weeks. The raters were not provided any rating scale to evaluate the essay since none is provided by the Boards they work with. They were asked to evaluate the essays as they did in the real in the examination centers and give an overall score to each essay. Only the seasoned raters with a minimum of ten years experience were recruited for this study to control for the ‘experience’ variable. There were 5 essays on each of the four topics (A Picnic Party, Patriotism, Co-education and Science). The essay topics were selected from the previous examination questions. After all the markers (n=94) had examined the same essay set (n=20) the second time (Time 2) they were asked to write a short written commentary to explain their evaluation criteria .The quantity of the response varied among individual and groups. Out of 473 paragraphs, 265 were written by males and 208 by female respondents. Following this task, 20 raters were interviewed using semi-structured format to crosscheck and better understand the findings from the qualitative as well as quantitative data.