BACKGROUND The Institute of Medicine has identified the comparative effectiveness of renal replacement therapies as a kidney-related topic among the top 100 national priorities. Given the importance of ensuring internal and external validity, the goal of this study was to identify potential sources of bias in observational studies that compare outcomes with different dialysis modalities. METHODS This observational cohort study used data from the electronic medical records of all patients that started maintenance dialysis in the calendar years 2007-2011 and underwent treatment for at least 60 days in any of the 2217 facilities operated by DaVita Inc. Each patient was assigned one of six dialysis modalities for each 91-day period from the date of first dialysis (thrice weekly in-center hemodialysis (HD), peritoneal dialysis (PD), less-frequent HD, home HD, frequent HD and nocturnal in-center HD). RESULTS Of the 162 644 patients, 18% underwent treatment with a modality other than HD for at least one 91-day period. Except for PD, patients started treatment with alternative modalities after variable lengths of treatment with HD; the time until a change in modality was shortest for less-frequent HD (median time = 6 months) and longest for frequent HD (median time = 15 months). Between 30 and 78% of patients transferred to another dialysis facility prior to change in modality. Finally, there were significant differences in baseline and time-varying clinical characteristics associated with dialysis modality. CONCLUSIONS This analysis identified numerous potential sources of bias in studies of the comparative effectiveness of dialysis modalities.