Previous studies have suggested that physical activity (PA) levels and cardiorespiratory fitness (CRF) impact on the autonomic control of heart rate (HR). However, previous studies evaluating PA levels did not discriminate between incidental PA and regular exercise. We hypothesized that incidental PA "per se" would influence cardiac autonomic indices as assessed via HR variability (HRV) and HR recovery (HRR) in non-exercisers. Thus, the objective of this study was to investigate the relationships between objective PA levels, CRF, and cardiac autonomic indices in adult, regular non-exercising female workers. After familiarization with procedures and evaluation of body composition, 21 women completed a submaximal cycling test and evaluation of HRR on four different days. Resting (2-min seated and standing) and ambulatory (4-h) HRV were also recorded. Levels of PA were assessed by accelerometry over five consecutive days (i.e., Wednesday to Sunday). Maximum oxygen consumption (VO2max) was measured as an index of CRF. As reliability was low to moderate for most HR measures, relationships between these and PA and CRF were examined using the 4-day average measures. Significant correlations were identified between post-exercise HRR in the first min with various PA indices (daily moderate PA, daily vigorous PA, and the sum of vigorous and very vigorous daily PA). Additionally, VO2max was significantly correlated to HRV but not to HRR. The current results indicated that CRF was influential in enhancing HRV while incidental or non-exercise based PA was associated with greater autonomic reactivation in adult overweight women. Therefore, both CRF and non-exercise based PA contribute significant but diverse effects on cardiac health. The use of 4-day averages instead of single measures for evaluation of autonomic control of HR may provide a better indication of regular cardiac autonomic function that remains to be refined.