Measuring Teacher Quality

A perennial issue at every college and university is how to measure teacher quality. It is important because it directly influences decisions about retention, tenure, and promotion. Everyone complains about basing such decisions on end-of-course evaluations. This column will explore a recent study by Scott Carrell and James West [1], undertaken at the United States Air Force Academy (USAFA), that strongly suggests that such evaluations are even less useful than commonly believed and that the greatest long-term learning does not come from those instructors who receive the strongest evaluations at the end of the class.

The study authors chose to measure teacher effectiveness in Calculus I by examining value added in both Calculus I and Calculus II: comparing student performance on course assessments for each instructor after controlling for variables in student preparation and background that included academic background, SAT verbal and math scores, sex, and race and ethnicity. It is generally acknowledged that better teachers produce better results in their students, but this has only been extensively studied in elementary students, and even there it is not without its problems. The authors reference a 2010 study by Rothstein [2] that shows a strong positive correlation between the quality of fifth grade teachers and student performance on assessments taken in fourth grade, suggesting a significant selection bias: The best students seek out the best teachers. This may be even truer at the university level where students have much more control over who they take a class with. For this reason, Carrell and West were very careful to measure the comparability of the classes. At USAFA, everyone takes Calculus I, and there is little personal choice in which section to take, so such selection bias is less likely to occur. The authors also tested for and found no evidence of backward correlation, that the best Calculus II instructors were correlated with higher grades in Calculus I.

The authors had a large sample size with which to work, all of the students who took Calculus I from fall 2000 through spring 2007, over 10,000 students and 91 instructors. The faculty make-up at USAFA is unusual among post-secondary institutions. There is a small core of permanent faculty. Only 15% of Calculus I instructors held the rank of Associate or Full Professor, and only 31% held a doctorate in mathematics or a mathematical science. Most of the teaching is done by officers who hold a master’s degree and are doing a rotation through USAFA. The average number of years of teaching experience among all Calculus I instructors was less than four years. Because of this, there is tight control on these courses, which facilitates a careful statistical study. There are common syllabi and examinations. All instructors get to see the examinations before they are given so that there is opportunity, if an instructor so wishes, to “teach to the test,” emphasizing those parts of the curriculum that are known to be important for the assessment.

Positive responses to the following prompts all had positive influence on student performance in Calculus I, significant at the 0.05 level:

1. Instructor’s ability to provide clear, well-organized instruction.
2. Value of questions and problems raised by instructor.
3.  Instructor’s knowledge of course material.
4. The course as a whole.
5.  Amount you learned in the course.
6. The instructor’s effectiveness in facilitating my learning in the course.

Surprisingly, these evaluations of the Calculus I instructor all had negative impact on student performance in Calculus II, with responses 1, 2, and 6 significant at the 0.05 level.

On the other hand, faculty rank, highest degree, and years of teaching experience were negatively correlated with examination performance in Calculus I, but positively correlated with performance in Calculus II, with statistical significance for years of teaching experience for both the negative impact in Calculus I and the positive impact in Calculus II.

The suggested implication is that less experienced instructors tend to focus on the particular skills and abilities needed to succeed in the next assessment and that students like that approach. Experienced instructors may pay more attention to the foundational knowledge that will serve the student in subsequent courses, and students appear to be less immediately appreciative of what these instructors are able to bring to the class.

This study strongly suggests that end of course student evaluations are, at best, an incomplete measure of an instructor’s effectiveness. It also suggests a long-term weakness of simply preparing students for their next assessment, though it should be emphasized that this represents merely a guess as to why less experienced instructors appear to get better performance from their students in Calculus I assessments.

At Macalester College, we recognize the importance of student reflection on what they learned months or years earlier. When a promotion or tenure case comes to the Personnel Committee, we collect online student evaluations of that faculty member from all of the students who have taken a course with him or her over roughly the past five years, combining both recent and current appraisals with longer term assessments of the effect that instructor has had.

References:

[1] Scott E. Carrell & James E. West, 2010. "Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors," Journal of Political Economy, University of Chicago Press, vol. 118(3), pages 409-432, 06. Available at http://www.nber.org/papers/w14081

[2] Jesse Rothstein, 2010. “Teacher Quality in Educational Production: Tracking, Decay, and Student Achievement.” Quarterly Journal of Economics 125 (1): 175–214. Available at  http://gsppi.berkeley.edu/faculty/jrothstein/published/rothstein_vam_may152009.pdf