The association of physical activity with the development and progression of coronary artery calcium (CAC) scores has not been studied. This study aimed to evaluate the prospective association between physical activity and CAC scores in apparently healthy adults. Prospective cohort study of men and women free of overt cardiovascular disease who underwent comprehensive health screening examinations between 1 March 2011 and 31 December 2017. Baseline physical activity was measured using the International Physical Activity Questionnaire Short Form (IPAQ-SF) and categorised into three groups (inactive, moderately active and health-enhancing physically active (HEPA)). The primary outcome was the difference in the 5-year change in CAC scores by physical activity category at baseline. We analysed 25 485 participants with at least two CAC score measurements. The proportions of participants who were inactive, moderately active and HEPA were 46.8%, 38.0% and 15.2%, respectively. The estimated adjusted average baseline CAC scores (95% confidence intervals) in participants who were inactive, moderately active and HEPA were 9.45 (8.76, 10.14), 10.20 (9.40, 11.00) and 12.04 (10.81, 13.26). Compared with participants who were inactive, the estimated adjusted 5-year average increases in CAC in moderately active and HEPA participants were 3.20 (0.72, 5.69) and 8.16 (4.80, 11.53). Higher physical activity was association with faster progression of CAC scores both in participants with CAC=0 at baseline and in those with prevalent CAC. We found a positive, graded association between physical activity and the prevalence and the progression of CAC, regardless of baseline CAC scores.