OBJECTIVE Validated algorithms (VAs) in insurance claims databases are often used to estimate the prevalence and incidence of comorbidities and evaluate safety signals. However, although they are then used in different data sources or subpopulations from those in which they were developed the replicability of these VAs are rarely tested, making their application and performance in these settings potentially unknown. This paper describes testing multiple VAs used to identify incident breast cancer cases in a general population and in an indication-specific population, patients with atopic dermatitis (AD). METHODS Two algorithms were tested in multiple insurance claims databases and four cohorts were created. Modifications were made to account for the US insurance setting. The resulting incidence rates (IRs) were then compared across algorithms and against surveillance, epidemiology, and end results (SEER) estimates to assess reliability. RESULTS Algorithm 1 produced low IRs compared to Algorithm 2. Algorithm 2 provided similar estimates to those of SEER. Individuals in the AD cohorts experienced lower incident breast cancer cases than those in the general population cohorts. CONCLUSIONS Regardless of an algorithm's reported accuracy, the original study setting and targeted population for the VAs may matter when attempting to replicate the algorithm in an indication-specific subpopulation or varying data sources. Investigators should use caution and conduct sensitivity analyses or use multiple algorithms when attempting to calculate incidence or prevalence estimates using VAs.