The aim of this study was to explore the reliability, validity, and factor structure of a newly developed Cognitive Behavioral Rating Scale (CBRS), designed to assess cognitive function through observation. The study included 180 stroke patients and 58 orthopedic patients. The CBRS and the Functional Independence Measure (FIM) were administered. Data on discharge disposition were also collected. The inter-rater reliability, construct validity, concurrent validity, predictive validity, and factor structure of the CBRS were evaluated. The interclass correlation coefficient was 0.736 (P<0.0001). Cronbach's α for the CBRS was 0.92. Patients with stroke showed a significantly higher total CBRS score than those with orthopedic problems (P<0.0001). Spearman's rank correlation coefficient (rs) was -0.70 (P<0.01) for the total FIM score and the CBRS, and -0.72 (P<0.01) for the cognitive FIM and the CBRS. The return-to-home group had a significantly lower CBRS total score than the not-to-home group (P<0.01). The CBRS comprised three factors: (i) 'language and performance', (ii) 'control of performance', and (iii) 'space-self relation.' The CBRS appears to have a high reliability and validity for the assessment of cognitive function.