Biliary tract cancers (BTCs) are rare but deadly cancers (gallbladder cancer [GBC], intrahepatic cholangiocarcinoma [ICC], extrahepatic cholangiocarcinoma [ECC], and ampulla of Vater cancer [AVC]). A recent US study reported increasing GBC incidence among people younger than 45 years and blacks; however, it did not examine trends for other biliary tract sites. This study characterized demographic differences in BTC incidence rates and time trends by anatomic site. Population-based North American Association of Central Cancer Registries data were used to calculate age-adjusted incidence rates, incidence rate ratios (IRRs), and estimated annual percent changes (eAPCs) for 1999-2013 by site and demographic group. For sites with significant differences in eAPC by age group, IRRs were compared by age group. GBC incidence rates declined among women (eAPC, -0.5%/y; P = .01) and all racial/ethnic groups except for non-Hispanic blacks, among whom rates increased (1.8%/y; P < .0001). Although GBC rates increased among 18- to 44-year-olds (eAPC, 1.8%/y; P = .01), they decreased among people 45 years old or older (-0.4%/y; P = .009). Sex (P < .0001) and racial/ethnic differences (P = .003 to .02) in GBC incidence were larger for younger people than older people. During this period, ICC (eAPC, 3.2%/y; P < .0001) and ECC rates (1.8%/y; P = .001) steadily increased across sex and racial/ethnic groups. Although AVC incidence rates increased among younger adults (eAPC, 1.8%/y; P = .03) but not older adults (-0.20%/y; P = .30), sex and racial/ethnic IRRs did not differ by age. Differential patterns of BTC rates and temporal trends have been identified by anatomic site and demographic groups. These findings highlight the need for large pooling projects to evaluate BTC risk factors by anatomic site.