The importance of cardiorespiratory fitness versus adiposity in determining heart rate variability (HRV) is unclear. From the Coronary Artery Risk Development in Young Adults study, an observational cohort study, we included 2316 participants (mean age = 45.2 ± 3.6 yr at year 20, 57% female, 43% Black) with HRV measured in 2005-2006 (year 20) and graded exercise test duration (GXTd) and adiposity measures (body mass index and waist circumference) obtained in 1985-1986 (baseline) and 2005-2006. HRV measures (SD of all normal RR intervals [SDNN] and square root of the mean value of the squares of differences between all successive RR intervals [RMSSD]) were obtained from resting 30-s 12-lead ECG. Cross-sectional associations between GXTd, adiposity, and HRV were assessed at year 20. Longitudinal changes in GXTd and adiposity measures were categorized as ≥10% increase, <10% change (no change), or ≥10% decrease. We used multivariable logistic regression to assess associations of GXTd and adiposity measures with unfavorable versus more favorable HRV (lower 25th percentile vs upper 75th percentile). A 1-SD increment in GXTd was associated with 22% and 32% lower odds of unfavorable SDNN and RMSSD, respectively; associations remained significant after adjustment for adiposity. A 1-SD increment in adiposity measures was associated with 16%-28% higher odds of unfavorable RMSSD; associations were not significant after adjustment for GXTd. Compared with no change/increase in GXTd, longitudinal decrease in GXTd was significantly associated with 55% and 94% higher odds of unfavorable SDNN and RMSSD, respectively, at year 20. These associations remained significant after adjusting for adiposity. Cardiorespiratory fitness may be a stronger determinant of HRV than adiposity. Intervention studies are needed to better determine the differential effects of improved cardiorespiratory fitness versus weight loss on HRV.