A Systematic Approach to Interrater Reliability in Teacher Performance Evaluations

Teacher evaluations are routinely conducted across the United States for licensure and professional development supports. However, there is limited research on the interrater reliability of these evaluation assessment systems, despite federal recommendations (Graham et al., 2012). This research explores the systematic approach to interrater reliability utilized by the Early Educator Support (EES) Office in North Carolina. The EES Office supports the Birth-through-Kindergarten (B-K) teacher licensure of over 900 early educators in both private and public sectors. The evaluators employed undergo extensive trainings and hold a B-K license themselves. As part of the training, the evaluators undergo an interrater reliability activity that requires them to rate ten fictitious profiles, using the North Carolina Teacher Evaluation Process (NCTEP) Rubric. This research aims to understand the evaluator response process. In this study, Many Facets Rasch Models are used to understand evaluator patterns of strictness, leniency and potential bias based on the race of teacher profile. Additionally, two of the models are compared to understand the extent that these rater response patterns are exhibited in their real caseloads of actual early educators. In conclusion, the group of evaluators do show evidence of strictness, leniency, and bias, however it is mostly exhibited by a small number of individual evaluators. It is possible to use the results to inform the professional growth of these evaluators, so that all early educators served by the EES Office receive valid, fair, and reliable teacher evaluations. Furthermore, it depicts a systematic approach to interrater reliability that could be used by other evaluation systems across the country.