This article examines the reliability and validity of the modified essay question. An 18-item modified essay question was developed and pretested on a group of clinical experts. After the pretest, the modified essay question was administered to a group of 25 physical therapy students immediately before their second level orthopedic placement. A reliability index of .39 (coefficient alpha) was calculated for the test. In addition, measures of validity were obtained by correlating the modified essay question scores with multiple-choice question scores and the students' in-clinic reasoning scores. The results suggest that the reliability coefficient found in this study is comparable to the medical literature. Also, the results showed that the modified essay question demonstrated a greater correlation with the in-clinic clinical reasoning score compared with the parallel content multiple-choice examination. Furthermore, preliminary evidence suggests that the modified essay question may be a useful paper and pencil tool in evaluating clinical reasoning. Finally, several shortcomings of this study are discussed in conjunction with a direction for further study.