Identifying optimal number of raters and items for dependable student evaluation of teaching: evidence from generalizability theory

Abstract

Abstract

Introduction Student evaluation of courses and teaching is widely used in higher education, yet questions remain regarding the number of student raters and evaluation items required to achieve dependable results. This study applied Generalizability Theory (GT) to determine optimal rateritem configurations for students appraisal of courses and teaching across different class sizes. Methods The study employed a quantitative measurement design using GT and analysed secondary student evaluation data from a public university in Ghana. The final analytic sample comprised 2,553 students nested within 145 courses. Separate GT analyses were conducted for course appraisal and teaching evaluation using a two-facet partially nested random design, (r:p) × i, in which raters were nested within lecturers and items were crossed with lecturers. Results Dependability increased mainly with the number of student raters, while item-related variance contributed little measurement error. For large classes, dependable course appraisal was achieved with approximately 1318 items rated by at least 60 students, whereas teaching evaluation achieved high dependability with as few as 12 items and 20 student raters. In small classes, dependable course appraisal was difficult to attain even with longer instruments, while acceptable dependability for teaching evaluation required approximately 1325 student raters. Conclusion The findings highlight the importance of context-sensitive evaluation design and show that increasing the number of raters is more effective for improving dependability than increasing the number of evaluation items. The study provides evidence-based guidance for designing efficient and dependable student evaluation systems in higher education.