OBJECTIVE To assess the psychometric qualities of a method of resident physician evaluation by faculty. METHODS Multicenter study by seven Physical Medicine and Rehabilitation training programs. Faculty physicians observed residents in brief patient encounters or teaching sessions, rated specific competencies, and provided residents with immediate feedback. The resident observation and competency assessment form included competencies in patient care, professionalism, interpersonal and communication skills, systems-based practice, and practice-based learning and improvement. Residents and faculty rated satisfaction with the process. RESULTS Three hundred sixty-two ratings were completed on 88 different residents. Each resident received an average of 3.8 ratings across two academic years. Overall internal consistency reliability was high (0.98); reliability of the individual competencies ranged from 0.74 to 0.76. Item means were correlated with year of training for two skill sets, with higher means for more experienced residents. The majority of participants gave high ratings of satisfaction; correlation between satisfaction ratings of attending and resident physicians was 0.63 (P < 0.01). CONCLUSIONS The resident observation and competency assessment is a reliable method to assess resident skills in five of six general competencies. Construct validity of the tool is supported by the fact that faculty rated two skill sets higher for senior residents.