BACKGROUND Despite calls for competency based education, a dearth of validated instruments for measuring basic skills currently exists. We developed an instrument to assess competency in basic surgical skills in second-year medical students and tested it for psychometric reliability and validity. METHODS From a review of the literature, an instrument comprised of numerically scaled items was constructed. After initial tests, several items were divided to produce a final instrument more specific and more appropriate for providing feedback to students. The final instrument was empirically tested for reliability and validity. RESULTS The final 10-item instrument is presented here along with all of the empirical evidence including internal consistency reliability and interrater reliability, and content, criterion-related, and construct validity. Overall alpha reliability was 0.84 and interrater reliability was r = 0.83, P < 0.01 for the total scores. Factor analysis provided evidence of construct validity. CONCLUSIONS The instrument has psychometric properties adequate for use as one criterion for summative evaluation and is educationally practical enough to provide focused and detailed feedback for student improvement.