The purpose of this study was to assess the degree of consistency in student ratings of teacher effectiveness during the first year of medical school. Student ratings of teaching effectiveness represent a commonly used source of information that enters into the academic decision-making process. In medical school, student evaluations often represent a major source of information that is used in promotion and tenure decisions. It is essential that the precision of such ratings be ascertained so that decision-makers will know how much confidence to place in this source of information on teaching effectiveness. In this study, each member of a first-year medical school class was randomly assigned a two-digit identification number at the beginning of the spring semester, 1986. As the semester progressed students were asked to evaluate each full-time teacher in three major courses. Multiple instructors were utilized in each course (n = 10). Each teacher was evaluated immediately after lectures during the first (T1) and second (T2) halves of the course. Students evaluated the teacher a third time (T3) as part of the end-of-semester overall course evaluation. The teachers were evaluated on a short eight-item Likert-type scale that identified several key indicators of effective teaching. Students attached their anonymous identification numbers to individual ratings so that their responses could be matched in the analysis. The results indicate that medical students are only moderately consistent in the extent to which they evaluate teachers. This inconsistency varied by course and by instructors within courses.(ABSTRACT TRUNCATED AT 250 WORDS)