Rater Errors

Engelhard (1994) and Saal, Downey and Lahey (1980) stated there are four major categories of rater errors. The first tendency is towards severity and leniency which is a rater consistently giving lower or higher appraisal than a deserved performance. Engelhard (1994) mentions that it should be the best for raters to be continuum severity or leniency, if test takers are scored by raters who have much severity variance. Thus people can get higher scores than they deserve or some may get lower scores than they ought to have. This surely can affect the validity of the scoring. In addition to that, raters can differ in severity at different criteria in the scale. (Schaefer, 2008) For example, it may be relatively easier to get a score of four

This creates no real sense of consistency to the overall raters‘ ratings. Meanwhile the fourth rater-error category is related to the third that the benchmarks and rangefinders at the criteria relating to the extent to which ratings are able to discriminate different test takers into different performance levels. If the raters fail to differentiate the written responses of the test takers based on the scoring criteria, then the purposes of measurement are defeated. The four categories of the above errors are identified as cross-sectional that can happen at any raters at one particular

Fitzpatrick, Ercikan, Yen, and Ferrara (1998) conduct two investigation where exams of third, fifth, and eighth grade students of many of subject areas are re-scored after one year and find out that the absolute standardized mean differences are relatively small - in the range of one-tenth upto two-tenths of a standard deviation. One of the exceptions is in Writing of Grade 5, where the mean difference can be considered large. They also calculate the correlations of total scores in the first and second sets of ratings. Correlations are consistently the highest in Mathematics, and consistently the lowest in Writing. Pearson correlations for third, fifth, and eighth grade writing are 0.58, 0.59, and 0.72, respectively. In this investigation, however, the raters in the first and second round are not the same