Self-consistent peer ranking for assessing student work : dealing with large populations

Assessing large populations of students puts a serious burden on teaching staff capacity. For open-format assignments, automation of the reviewing process can offer only limited support. Peer ranking is a partial solution to the problem, with the added benefit that students' critical reading skills are developed. We see two remaining problems, however: (1) for students, it is a major challenge to assign marks on an absolute scale, and (2) students' competence in reviewing may vary significantly - so not all peer reviews should have a similar weight in the process. To remedy these shortcomings, we suggest an approach to peer ranking, inspired by Jon Kleinberg's FHTS-algorithm, where both the students' assignment results and the quality of their double anonymous peer reviews are algorithmically ranked. Based on preliminary model calculations, we estimate that this strategy may reduce the required effort for reviewing open-format assignments approximately by a factor of ten. A first large-scale pilot with this method will take place in undergraduate courses at Eindhoven University of Technology, spring 2013. Since this involves about 900 students, automated support is a must. We describe the peer reviewing facilities that were introduced in our web-based education support system named Peach.