BACKGROUND Although depression has been regarded as a major public health problem, many individuals with depression still remain undetected or untreated. Despite the potential for Internet-based tools to greatly improve the success rate of screening for depression, their reliability and validity has not been well studied. Therefore the aim of this study was to evaluate the test-retest reliability and criterion validity of a Web-based system, the Internet-based Self-assessment Program for Depression (ISP-D). METHODS The ISP-D to screen for major depressive disorder (MDD), minor depressive disorder (MinD), and subsyndromal depressive symptoms (SSD) was developed in traditional Chinese. Volunteers, 18 years and older, were recruited via the Internet and then assessed twice on the online ISP-D system to investigate the test-retest reliability of the test. They were subsequently prompted to schedule face-to-face interviews. The interviews were performed by the research psychiatrists using the Mini-International Neuropsychiatric Interview and the diagnoses made according to DSM-IV diagnostic criteria were used for the statistics of criterion validity. Kappa (kappa) values were calculated to assess test-retest reliability. RESULTS A total of 579 volunteer subjects were administered the test. Most of the subjects were young (mean age: 26.2 +/- 6.6 years), female (77.7%), single (81.6%), and well educated (61.9% college or higher). The distributions of MDD, MinD, SSD and no depression specified were 30.9%, 7.4%, 15.2%, and 46.5%, respectively. The mean time to complete the ISP-D was 8.89 +/- 6.77 min. One hundred and eighty-four of the respondents completed the retest (response rate: 31.8%). Our analysis revealed that the 2-week test-retest reliability for ISP-D was excellent (weighted kappa = 0.801). Fifty-five participants completed the face-to-face interview for the validity study. The sensitivity, specificity, positive, and negative predictive values for major depressive disorder were 81.8% and 72.7%, 66.7%, and 85.7% respectively. The overall accuracy was 76.4%. CONCLUSIONS The evidence indicates the ISP-D is a reliable and valid online tool for assessing depression. Further studies should test the ISP-D in clinical settings to increase its applications in clinical environments with different populations and in a larger sample size.